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Foreword 


ORGANIZATION 


The  Human  Factors  and  Medicine  (HFM)  Research  Technology  Organisation  (RTO)  Task  Group  (RTG)  129 
began  in  2004  as  Exploratory  Team  (ET)  046.  The  HFM  Panel  (HFM)  approved  its  Technical  Activity  Proposal 
(TAP)  and  the  Terms  Of  Reference  (TOR)  at  the  Executive  Session  of  the  Research  Technology  Board  in  March 
2006.  HFM- 129  held  four  meetings.  The  TAP  and  the  TOR  are  given  in  Annex  A  and  B,  respectively.  Annex  C 
lists  all  the  papers,  presentations,  and  compact  disks  that  resulted  from  work  done  in  HFM-129.  Annex  D  lists  the 
principal  activities  of  Task  Group  HFM-129. 
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Advanced  Authoring  Technologies, 
Capabilities  and  Opportunities 

(RT  O-TR-HFM- 129) 

Executive  Summary 

Today’s  volatile  operating  environments  make  it  increasingly  important  for  learning  environments 
(e.g.,  computer-assisted  instruction,  interactive  multi-media  instruction,  Modeling  and  Simulation  (M&S) 
for  mission  rehearsal  and  training,  and  simulation-based  training  for  equipment  operation,  maintenance, 
and  deployment)  to  adapt  quickly  to  rapidly  changing  and  unexpected  operational  requirements.  Human 
Factors  and  Medicine  (HFM)-129  was  tasked  to  assess  the  use  of  software  technologies  and  opportunities 
in  “authoring”  -  designing,  developing,  and  maintaining  computer-mediated  learning  environments 
rapidly,  accurately,  and  efficiently  while  sustaining  -  if  not  enhancing  -  their  relevance  and  quality. 

As  used  in  this  report,  authoring  is  a  generic  term  primarily  covering  three  applications: 

1)  Authoring  languages:  Notation  systems  for  capturing  and  describing  the  design  and  functional 
characteristics  of  a  learning  environment  in  a  way  that  computer  systems  can  interpret  correctly 
and  comprehensively. 

2)  Authoring  systems:  Suites  of  functional  capabilities  intended  to  assist  instructional  designers  in 
implementing  some  or  all  components  of  instruction  system  design  (i.e.,  analysis,  design,  production, 
implementation/delivery,  and  evaluation). 

3)  Authoring  tools:  Modular  components  of  an  authoring  language  and/or  an  authoring  system. 

Authoring  has  been  a  concern  of  computer-mediated  instruction  since  its  inception  in  the  1950s.  It  is 
intended  to  make  more  efficient  (faster,  more  effective,  and  less  expensive)  the  development  of  learning 
environments  by  subject  matter  specialists  who  have  a  limited  background  or  training  in  computer 
technology.  Over  time,  authoring  capabilities  have  been  increasingly  infused  with  multimedia,  integrated 
with  processes  intended  to  track,  guide,  and  assess  learning,  modularized  to  support  tailoring  to  the  needs 
of  individual  learners,  standardized  to  enable  interoperability  and  reuse,  and  made  more  accessible, 
flexible,  and  easy  to  use.  Tradeoffs  between  ease  of  use  and  flexibility  of  use  are  keys  in  enhancing  the 
power  of  authoring  capabilities. 

The  Task  Group  (TG)  found  and  discussed  examples  of  advanced  authoring  capabilities  in  four  primary  areas: 

1)  Rapid,  agile  development  of  mission  training  simulations  by  local  commands; 

2)  Efficient,  cost-effective  development  of  simulations  for  operating  and  maintaining  equipment; 

3)  Semi-automated  generation  of  instructional  material  from  standard  office  applications  such  as  word 
processing  and  projection  slide  tools;  and 

4)  Peer  authoring  using  Web  2.0  technologies. 

The  TG  found  several  publications  that  discuss  the  selection  of  authoring  capabilities  and  products. 
Twenty-six  questions  were  developed  by  the  TG  to  summarize  issues  raised  by  these  publications  and  to 
act  as  guidelines  for  developers  in  NATO  and  Partner  countries  to  use  in  selecting  among  current  and 
emerging  authoring  capabilities. 
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Technologies  avancees  d’authoring, 
possibility  et  opportunity 

(RT  O-TR-HFM- 129) 


Synthese 

A  l’heure  actuelle,  compte  tenu  de  la  versatility  de  l'environnement  operationnel  il  est  de  plus  en  plus 
important,  pour  l’environnement  pedagogique  (par  exemple,  l'enseignement  assiste  par  ordinateur, 
l’instruction  multimedia  interactive,  la  modelisation  et  la  simulation  (M&S)  pour  l’entrai'nement  ou  pour 
la  repetition  de  mission,  et  la  formation  basee  sur  la  simulation  pour  le  fonctionnement  des  equipements, 
leur  maintenance  et  leur  deployment)  de  s’ adapter  rapidement  a  des  exigences  operationnelles  imprevues 
et  changeantes.  Le  groupe  de  travail  sur  les  facteurs  humains  et  la  medecine  (Human  Factors  and  Medicine 
-  HFM)-129  a  ete  charge  d’evaluer  l’utilisation  des  technologies  logicielles  ainsi  que  les  opportunity 
en  matiere  d’  «  authoring  »,  de  conception,  de  developpement,  et  de  maintenance  des  environnements 
pedagogiques  informatises,  de  fagon  rapide,  precise  et  efficace  tout  en  maintenant  -  si  ce  n’est  en  ameliorant  - 
leur  pertinence  et  leurs  qualites. 

Dans  ce  rapport,  authoring  est  un  terme  generique  couvrant  principalement  trois  applications  : 

1)  Langages  d’authoring  :  Systemes  de  notation  pour  la  capture  et  la  description  de  la  conception  et 
des  caracteristiques  fonctionnelles  d’un  environnement  d’ etude  de  telle  fagon  que  des  systemes 
informatises  puissent  1’ interpreter  correctement  et  de  maniere  exhaustive. 

2)  Systemes  d’authoring  :  Series  de  capacites  fonctionnelles  destinees  a  assister  les  concepteurs 
d’enseignement  pour  la  mise  en  oeuvre  de  quelques  uns  ou  de  tous  les  composants  de  la  conception 
d’un  systeme  destruction  (par  exemple,  l’analyse,  la  conception,  la  production,  1’ implantation/ 
livraison,  et  revaluation). 

3)  Outils  d’authoring  :  Composants  modulaires  d’un  langage  d’authoring  et/ou  d’un  systeme 
d’authoring. 

L’authoring  s’est  interesse  a  l’enseignement  assiste  par  ordinateur  depuis  sa  creation  dans  les  annees  50. 
Le  but  est  de  rendre  plus  efficient  (plus  rapide,  plus  efficace,  et  moins  cher)  le  developpement 
d’ environnements  pedagogiques  par  des  specialistes  du  sujet,  qui  ont  des  connaissances  ou  une  formation 
limitees  en  informatique.  Au  fil  du  temps,  les  capacites  de  l’authoring  ont  ete  de  plus  en  plus  liees  aux 
multimedias,  integrees  dans  des  processus  prevus  pour  suivre,  guider  et  evaluer  l'enseignement, 
modularisees  pour  s’adapter  aux  besoins  d’eleves  isoles,  standardisees  pour  permettre  l’interoperabilite  et 
la  reutilisation,  et  rendues  plus  accessibles,  plus  flexibles,  et  faciles  d’utilisation.  Les  compromis  entre  la 
facilite  d’utilisation  et  la  flexibilite  sont  les  cles  de  l’amelioration  des  possibilites  de  l’authoring. 

Le  groupe  de  travail  (TG)  a  trouve  et  discute  d’exemples  de  capacites  avancees  d’authoring  dans  quatre 
domaines  principaux  : 

1)  Developpement  rapide,  et  souple  de  simulations  d’entrainement  de  mission  par  les  commandements 
locaux  ; 

2)  Developpement  de  simulations  efficaces  et  rentables  pour  le  fonctionnement  et  la  maintenance  des 
equipements  ; 
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3)  Generation  semi-automatisee  de  materiels  d’instruction  provenant  d’applications  de  materiels  de 
bureau  standards  comme  les  outils  de  traitement  de  texte  et  de  projection  de  diapositives  ;  et 

4)  Authoring  par  les  pairs  en  utilisant  les  technologies  Web  2.0. 

Le  TG  a  trouve  plusieurs  publications  qui  traitent  de  la  selection  des  capacites  et  des  produits  d’ authoring. 
Trente  six  questions  ont  ete  developpees  par  le  TG  pour  resumer  les  problemes  souleves  par  ces 
publications  et  pour  en  faire  des  directives  a  Tintention  des  developpeurs  de  l’OTAN  et  des  nations 
partenaires  afm  qu’elles  les  utilisent  dans  leur  choix  d’ authoring  actuels  et  emergents. 
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Chapter  1  -  BACKGROUND 


ORGANIZATION 


NATO  Task  Groups  (TGs)  are  established  by  Technical  Panels  operating  under  the  NATO  Research  and 
Technology  Organisation  (RTO)  and  facilitated  by  the  NATO  Research  and  Technology  Agency  (RTA). 
The  NATO  RTO  promotes  and  conducts  cooperative  scientific  research  and  exchange  of  technical  information 
among  26  NATO  nations  and  38  NATO  partners.  TGs  are  established  by  Technical  Panels  and  charged  to 
perform  specific  research  activities  within  a  specific  amount  of  time. 

TG  Human  Factors  and  Medicine  (HFM)-129,  “Advanced  Authoring  Technologies,  Capabilities,  and 
Opportunities,”  was  established  in  2005  by  the  RTO’s  HFM  Panel  (HFMP).  The  FLFMP’s  mission  is  to  provide 
the  Science  and  Technology  (S&T)  base  for  optimizing  health,  human  protection,  well-being,  and  performance 
of  humans  in  operational  environments.  This  mission  involves  understanding  and  ensuring  the  physical, 
physiological,  psychological,  and  cognitive  compatibility  of  military  personnel.  It  is  accomplished  by  exchange 
of  information,  collaborative  experiments,  and  shared  field  trials.  The  principal  activities  of  Task  Group 
HFM- 129  are  provided  in  Annex  D. 
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Today’s  volatile  operating  environments  make  increasingly  important  the  ability  of  computer-mediated 
training,  decision  aiding,  performance  support,  mission  rehearsal,  training  Modeling  and  Simulation  (M&S), 
and  related  capabilities,  which  may  be  generically  described  as  “learning  environments,”  to  adapt  quickly  to 
rapidly  changing  and  unexpected  operational  requirements.  HFM-129  was  tasked  to  assess  the  capabilities  and 
use  of  software  technologies  and  opportunities  for  “authoring”  -  designing,  developing,  and  maintaining 
computer-mediated  learning  environments  rapidly,  easily,  and  effectively.  An  important  consideration  was  to 
minimize  the  costs  and  other  resources  required  to  design,  develop,  and  maintain  these  environments  while 
sustaining,  if  not  enhancing,  their  relevance  and  quality. 

Task  Group  HFM-129  identified  advanced  technologies  for  authoring,  demonstrated  how  these  technologies  can 
provide  NATO  forces  the  learning  environments  needed  to  meet  today’s  operating  requirements,  and  developed 
guidelines  (Chapter  7)  for  choosing  among  them.  By  increasing  the  agility,  relevance,  and  accessibility  of  these 
environments,  the  application  of  advanced  authoring  opportunities  and  capabilities  can  similarly  enhance  the 
operational  capabilities  and  performance  of  NATO  units. 
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Authoring  languages,  as  used  in  this  report,  are  technical  notation  systems  for  capturing  and  describing  the 
design  and  functional  characteristics  of  a  learning  environment  so  that  computer  systems  can  interpret  these 
characteristics  correctly  and  comprehensively.  Authoring  languages  are  a  special  instance  of  higher  level 
programming  languages.  They  are  often  intended,  with  greater  and  lesser  success,  to  allow  individuals  who 
are  not  computer  specialists  to  prepare  materials  for  computer  presentation.  While  authoring  languages  provide 
easy,  effective  use  by  classroom  instructors,  many  instructors  also  use  more  generic  commercial  products  such 
as  Dreamweaver,  Flash,  and  PowerPoint  as  authoring  capabilities  to  create  computer-based  materials  for  their 
students. 

Authoring  tools  are  made  available  either  as  modular  components  of  an  authoring  language  or  system  or  are 
provided  for  use  in  a  stand-alone  mode. 

An  authoring  system  is  a  suite  of  functional  capabilities  intended  to  assist  instructional  designers  in 
implementing  all  the  components  of  instruction  system  design  -  that  is,  analysis  (to  identify  instructional 
objectives),  design  (to  achieve  instructional  objectives  efficiently  and  effectively),  production  (to  accurately 
render  content  and  design  in  operational  software),  implementation/delivery  (to  establish  computer-mediated 
learning  environments),  and  evaluation  (to  ensure  the  effectiveness,  efficiency,  and  relevance  of  the  learning 
environment).  Most  authoring  capabilities  concentrate  on  design  and/or  production  of  materials  for 
implementation. 

Authoring  capabilities  is  a  generic  term  used  in  this  report  to  cover  all  three  of  these  applications. 
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As  books  by  Galanter  [1]  and  Coulson  [2]  attest,  computer  technology  has  been  used  to  create  learning 
environments  since  the  1950s.  Sufficient  empirical  data  on  the  effectiveness  of  these  environments  had  been 
amassed  by  the  early  1970s  to  indicate  their  considerable  promise  and  value  for  education,  training, 
and  performance  aiding  (e.g.,  [3],  [4],  [5],  [6],  [7]).  However,  two  significant  obstacles  to  widespread 
implementation  and  adoption  of  these  environments  were  found: 

•  The  cost  of  computer  technology  itself;  and 

•  The  cost  of  preparing  (“authoring”)  learning  materials  for  computer  control  and  delivery. 

Moore’s  Law  (e.g.,  [8])  appears  to  be  removing  the  first  of  these  problems.  It  suggests  that  the  number  of 
electronic  devices  (e.g.,  transistors)  on  chips  is  doubling  about  every  18  months.  This  pace  of  development 
seems  likely  to  continue.  Gorbis  and  Pescovitz  [9]  found  that  about  70%  of  Institute  for  Electrical  and 
Electronic  Engineers  (IEEE)  Fellows  expect  Moore’s  Law  to  continue  holding  for  at  least  10  more  years. 
About  35%  of  them  expect  it  to  continue  beyond  that,  up  to  20  years. 

The  second  obstacle  remains.  Anticipating  every  possible  instructional  interaction  and  every  possible  state  of 
every  possible  learner  and  then  incorporating  all  these  possibilities  into  computer  programs  that  organize  and 
manage  them  are  still  expensive  and  time-consuming.  One  way  around  this  problem  is  simply  to  ignore  it  and 
prepare  page-turning  instruction  that  allows  learners  to  progress  through  material  by  successively  typing  a 
“next”  key.  Because  computer-mediated  instruction  can  deliver  education  and  training  to  large  numbers  of 
learners  at  minor  incremental  costs,  there  are  instances  in  which  this  approach  may  be  better  than  residential, 
classroom  instruction.  However,  it  has  been  found  to  be  less  than  optimally  successful,  and  it  fails  to  employ 
the  capabilities  of  our  current  and  rapidly  developing  computer  technology  and  our  abilities  to  apply  it  [10]. 

Another  way  to  solve  the  problem  relies  on  what  in  the  1970s  was  called  Intelligent  Computer  Assisted 
Instruction  (ICAI)  and  is  now  typically  referred  to  as  Intelligent  Tutoring  Systems  (ITS).  This  approach  was 
partially  motivated  by  developments  in  artificial  intelligence  but  more  particularly  by  the  promise  of  reducing 
or  even  eliminating  authoring  costs  by  enabling  the  computers  to  generate  and  assemble  in  a  near 
conversational  manner  the  instructional  interactions  of  one-on-one  tutoring,  on-demand  and  in  real  time. 

The  key  to  this  approach  was  envisioned  by  Uttal  [11]  and  Carbonell  [12]  as  imbuing  computers  with  the 
conversational,  mixed-initiative,  instructional  dialogue  practiced  by  human  tutors.  Work  in  this  area  has 
continued  from  the  late  1960s  to  the  present  (e.g.,  [13],  [14],  [15],  [16]).  It  is  a  key  goal  of  current,  influential 
efforts  such  as  the  Advanced  Distributed  Learning  (ADL)  initiative  [17]  and  the  Defense  Advanced  Research 
Projects  Agency’s  (DARPA)  Education  Dominance  program.  However,  despite  notable  progress,  the  goal  of 
generating  instructional,  tutorial  dialogues  on-demand  and  in  real  time  remains  elusive. 

We  are  left,  then,  with  authoring  systems,  languages,  and  tools  in  the  hands  of  human  developers. 
Like  computer-assisted  instruction  itself,  authoring  capabilities  have  been  present  almost  since  the  beginning 
of  computer  technology.  The  1950s  System  Training  Program  was  used  to  prepare  technicians  to  operate  the 
Semi-Automated  Ground  Environment  (SAGE),  which  tracked  aircraft  flying  over  the  North  Pole  into  North 
American  airspace.  It  included  a  variety  of  computer  programs  to  help  design  and  develop  training  exercises 
[18].  The  many  developers  of  computer-assisted  instruction  who,  in  the  late  1950s  and  through  most  of  the 
1960s,  were  programming  in  assembler  language  and  disdained  the  use  of  higher-order  languages,  including 
authoring  languages,  developed  libraries  of  sub-routines  that  were,  in  effect,  authoring  tools  used  to  speed  up 
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and  simplify  the  authoring  process  [19].  Because  programmers  of  all  sorts  were  expensive,  few  in  number, 
and  seldom  aware  of  instructional  design  techniques  and  practices,  the  computer  languages,  tools,  and  systems 
specifically  designed  for  use  by  instructional  technologists  in  authoring  materials  increasingly  began  to 
appear. 

4.1  AUTHORING  LANGUAGES,  TOOLS,  AND  SYSTEMS 

The  motivation  behind  the  development  of  authoring  languages,  then,  was  to  enable  classroom  instructors, 
instruction  designers,  and  other  specialists  in  instruction  to  do  what  computer  programmers  were  doing  but 
without  requiring  an  extensive  knowledge  of  computer  programming  or  operations.  Early,  widely  used  examples 
of  full-purpose  authoring  languages  were  COURSEWRITER,  developed  for  the  IBM  1500  Computer- Assisted 
Instruction  System  [20],  and  TUTOR,  developed  for  the  University  of  Illinois  Programmed  Logic  for  Automated 
Teaching  Operations  (PLATO)  [21]. 

COURSEWRITER  was  a  macro-oriented  language  that  required  designers  to  fill  in  the  blanks  of  templates  for 
macros  that  were  then  expanded  into  computer  code.  It  was  designed  to  allow  elementary  and  high  school 
classroom  teachers  to  prepare  computer-assisted  instruction  on  the  then  state-of-the-art  IBM  1500  system. 
Using  it,  they  were  expected  to  develop  materials  using  random  access  to  audio  (stored  on  analog  tapes), 
audio  message  recording,  random  access  to  photographs  (stored  on  film  strips),  text  displayed  on  cathode-ray 
tube  displays,  light  pens,  and  keyboard  input.  As  it  turned  out,  mid-level  programmers  had  to  be  hired  to  do 
this  work. 

TUTOR  was  closer  to  a  typical  higher  order  computer  programming  language  than  COURSEWRITER. 
It  was,  again,  intended  to  help  classroom  instructors  (but  generally  at  the  college  and  university  level)  create 
materials  for  delivery  on  the  University  of  Illinois  PLATO  computer  system.  A  prominent  feature  of  that  system 
was  its  plasma  panel,  which  presented  digitized  graphics  (including  some  primitive  animation)  to  students,  along 
with  instructional  text  during  the  system’s  “teaching  operations”. 

Both  of  these  authoring  languages  assumed  an  instructional  approach  based  on  a  combination  of  Keller’s  [22] 
Personalized  System  of  Instruction  (PSI)  and  Crowder’s  [23]  intrinsic  programming.  These  approaches  are 
characteristic  of  much  of  the  computer-assisted  instruction  designed  and  presented  today  and,  more  to  the  point, 
are  characteristic  of  the  approaches  many  developers  assume  in  designing  authoring  languages  and  tools. 

With  PSI,  Keller  articulated  the  common  instructional  approach  of  dividing  instructional  programs  into  units, 
usually  presented  in  a  linear  sequence,  and  then  requiring  students  to  demonstrate  mastery  of  a  unit  before 
allowing  them  to  proceed  to  the  next  unit.  Pre-testing  before  entering  a  unit  determines  if  a  learner  needs  to 
enter  it  at  all,  and  post-testing  determines  if  the  learner  has  mastered  the  material  in  the  unit  after  passing 
through  it. 

A  comprehensive  review  of  PSI  effectiveness  by  Kulik,  Kulik,  and  Cohen  [24]  found  that  PSI  programs  raised 
final  examination  scores  by  about  0.50  standard  deviations  over  programs  using  conventional  (non-PSI) 
means  of  classroom  instruction.  They  also  found  that  PSI  produced  less  variation  in  achievement,  higher 
student  ratings,  and  fewer  course  withdrawals  across  a  variety  of  subject  matters  and  course  settings. 
However,  Keller,  writing  in  1985  [25],  was  pessimistic  about  the  future  of  PSI.  He  cited  the  large  investment 
of  instructor  time  needed  to  set  up  PSI  courses  and  the  general  lack  of  support  from  administrators  to 
compensate  instructors  for  that  time.  Today,  computer  technology  has  significantly  reduced  the  labor  needed 
to  use  PSI,  and  it  now  appears  to  be  a  practicable  instructional  approach. 
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If  PSI  is  a  viable  approach  for  managing  instruction  modularized  into  units,  what  approach  might  be  used  for 
instructional  management  within  these  units?  Today,  many  instructional  applications  use  “intrinsic 
programming”  for  this  purpose.  First  described  by  Crowder  [23]  and  commonly  used  in  programmed  textbooks 
and  tutorial  computer-based  instruction,  intrinsic  programming  is  a  practical,  easily  understood  approach  that 
emphasizes  active  responding  and  immediate  feedback  to  learners.  By  using  responses  made  by  learners  to 
determine  their  next  steps,  it  allows  for  more  individualization  than  simply  allowing  them  to  set  their  own  pace 
through  linear  material. 


The  following  sample  instructional  item,  adopted  from  one  provided  by  Crowder,  is  typical  and  its  approach 
may  appear  familiar: 

In  the  multiplication  3x4=12,  the  number  12  is  called  a _ . 


A) 

Factor 

B) 

Quotient 

C) 

Product 

D) 

Power 

In  this  item,  the  system  assumes  that  a  student  responding  with  “A”  misunderstands  the  meaning  of  “Factor” 
and/or  lacks  an  understanding  of  “Product”.  The  student  is  then  branched  to  some  instructional  materials 
intended  to  correct  one  or  both  of  these  cognitive  states  and  returned  to  this  item  or  one  similar  to  it.  The  same 
type  of  remedial  approach  is  applied  to  responses  of  “B”  and  “D”.  A  student  responding  with  “C”  may  be 
rewarded  with  positive  feedback  and  sent  on  to  whatever  item  continues  progress  toward  the  instructional 
goal(s),  an  action  that  by  itself  may  constitute  positive  reinforcement. 

Crowder’s  intrinsic  programming  can  be  found  posing  questions  and  providing  individualized  branching 
following  text  paragraphs,  graphic  displays,  simulations,  audio  presentations,  video  sequences,  and/or  other 
sources  of  instructional  content.  The  underlying  logic  remains  the  same:  display  something,  elicit  a  response, 
and  branch  to  remedial  or  reinforcing  material  depending  on  the  response. 

We  can  assume  the  application  of  cognitive  models  to  be  a  recent  innovation  in  technology-based  instruction. 
However,  intrinsic  programming,  which  has  been  in  wide  use  since  the  1960s,  requires  instructional  designers 
to  anticipate  all  wrong  responses  likely  to  be  made  by  learners,  the  misconceptions  and  other  sources  of  these 
wrong  responses,  and  the  remedial  material  needed  to  correct  them.  This  approach  requires  an  underlying, 
albeit  implicit,  model  of  cognition  and  instructional  progress  that  can  be  used  to  assess  transitions  from 
unlearned  to  learned  states  in  the  subject  matter  presented.  To  the  extent  that  this  model  is  valid,  it  can  support 
individualization  of  pace,  content,  sequence,  and  style,  and  authoring  because  it  is  fairly  straightforward. 

In  a  review  of  57  studies,  Kulik,  Cohen,  and  Ebeling  [26]  found  that  this  type  of  programmed  instruction,  used 
in  higher  education  to  present  a  variety  of  subject  areas,  improved  performance  by  about  0.24  standard 
deviations  over  conventional  instruction.  These  findings  (among  others)  suggest  that  the  positive  impact  of 
intrinsic  programming  is  genuine,  if  limited. 

Today,  much  computer-assisted  instruction  uses  modularization  to  combine  Keller’s  and  Crowder’s 
approaches,  as  shown  in  Figure  4-1.  Many  authoring  languages  and  tools,  starting  with  COURSEWRITER 
and  TUTOR,  have  been  designed,  with  greater  and  lesser  success,  to  allow  instructional  specialists  with  only 
rudimentary  computer  skills  to  implement  this  instructional  approach. 
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From  previous  module  ^  „ 

- c - >  Pre-test 

w  ^  Yes,  go  to  next  module 

- Pass?  - 2 - 

No,  go  to  this  module X 


In  the  multiplication  of  3  x  4  =  12, 
the  number  12  is  called  a 

A. 

Factor 

[Branch  to  remedial  XI] 

B. 

Quotient 

[Branch  to  remedial  X2 

C. 

Product 

[Reinforce,  go  to  next] 

D. 

Power 

[Branch  to  remedial  X3 

Post-test 


Figure  4-1 :  Typical  Use  Case  -  Keller’s  PSI  Approach 
Combined  with  Crowder’s  Intrinsic  Programming. 
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The  previous  section  notes  that  the  development  of  instructional  software  is  a  costly  and  time-consuming 
process.  This  bottleneck  is  becoming  more  serious  because  current  trends  in  learning  technology,  such  as  just- 
in-time  and  just-enough  learning  [27]  increase  the  need  for  large  amounts  and  many  different  kinds  of 
instructional  software  to  be  created  for  very  specific  needs  in  very  short  periods  of  time. 

Because  most  knowledge  domains  targeted  for  instruction  are  specific,  instructional  software  needs  to  be 
developed  as  custom-made  software  by  domain  specialists.  These  specialists  already  possess  the  necessary 
domain  knowledge  and  have  easy  access  to  relevant  (i.e.,  multimedia)  resources  [28].  Moreover,  professional 
instructional  designers  and  software  producers  are  not  readily  available  and/or  affordable.  In  organizations 
such  as  the  military,  senior  domain  specialists  who  are  typically  expected  to  teach  others  about  their  domain  and 
to  design  traditional  instructional  materials  are  now  being  tapped  to  produce  computer-mediated  environments 
for  instruction. 

How  can  instructional  software  development  by  domain  specialists  be  made  more  efficient  (i.e.,  faster,  less 
costly,  easier)  and  more  effective  (i.e.,  improved  didactical  quality  of  the  final  product)?  One  way  is  to  provide 
formal  training  to  improve  both  design  and  production  skills  for  creating  this  software.  However  and  particularly 
in  the  military,  such  training  is  often  not  feasible  due  to  lack  of  budget,  time,  and/or  suitable  training  programs. 
Another  way,  as  suggested  in  the  previous  section,  is  to  provide  authoring  tools  that  facilitate  the  transition  from 
paper-based  instructional  blueprints  to  instructional  software  code,  making  the  process  faster,  easier,  and  less 
expensive.  Such  tools  encapsulate  and  avoid  the  complexities  of  computer  programming  by  providing  intuitive 
interfaces,  templates,  and  pre-structured  program  components. 

The  Task  Group  reviewed  authoring  capabilities  that  have  been  developed  in  six  countries:  Canada,  Denmark, 
Netherlands,  Norway,  the  United  Kingdom,  and  the  United  States.  No  meaningful  differences  in  authoring 
capabilities  were  found  across  these  countries.  Because  authoring  capabilities  are  intended  to  help  developers 
focus  on  instruction  techniques  rather  than  on  the  computer  technology  required  to  implement  them,  it  seems 
unlikely  that  their  functional  characteristics  would  vary  substantially  based  on  country  of  origin.  Three  NATO 
surveys  of  ADL  technologies  across  NATO  and  Partner  countries  did  not  reveal  any  substantial  differences  in 
purpose  or  design  [29],  [30],  [31].  NATO  and  Partner  countries  were  found  to  be  using  similar  authoring 
capabilities  successfully. 

Authoring  capabilities  have  originated  from: 

•  Instructional  design  (e.g.,  Authorware,  Course  Genie); 

•  Software  engineering  (e.g.,  Flash,  Captivate,  Office); 

•  Academia  (e.g.,  SimQuest,  Reusable  Educational  Design  Environment  and  Engineering  Methodology 
(REDEEM)); 

•  Learning  Content  Management  System  (LCMS)  vendors  (e.g.,  Desire2Leam,  Learn  Exact); 

•  M&S  (e.g.,  RAPIDS,  Experimental  Advanced  Instructional  Advisor  (XAIDA),  and  Diagnostic 
Instruction  and  Guidance  (DIAG); 

•  Expert  systems  (e.g.,  Demonstr8,  D3  trainer,  and  DIAG); 

•  Serious  gaming  (e.g.,  Ambush!,  Tactical  Language  and  Culture  Training  System  (TLCTS));  and 

•  Standardization  initiatives  (e.g.,  Reusable  eLearning  Object  Authoring  And  Delivery  (RELOAD)). 
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Vendors  often  claim  that  their  authoring  tools  allow  for  the  easy  and  rapid  creation  of  instructional  software 
with  a  high  didactical  quality  -  even  when  developers  have  little  or  no  experience.  To  various  degrees  their 
claims  are  justified. 

Over  all,  authoring  is  used  to: 

•  Increase  speed  of  preparation  (e.g.,  rapid  simulation/scenario  generation  -  authoring  and  editing); 

•  Increase  ease  of  preparation  -  everyone  an  author; 

•  Increase  “sharability”  (portability,  durability,  reusability,  accessibility); 

•  Increase  instructional  effectiveness; 

•  Increase  technical  reliability;  and 

•  Reduce  cost  of  preparation. 
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Authoring  capabilities  are  increasingly: 

•  Incorporating  Multimedia  -  They  are  including  interactive  video,  avatars,  simulated  entities,  dynamic 
terrain  effects,  three-dimensional  (3D)  sound,  and  so  forth. 

•  Integrated  -  They  allow  editing  to  span  a  larger  range  of  technical  and  educational  features,  such  as 
tracking  and  guiding  learning  processes,  integrating  a  range  of  media  with  advanced  simulations, 
and  providing  diagnostics. 

•  Modular  -  They  allow  selective  launching  or  disabling  of  learning  modules  to  tailor  the  environment 
for  individual  users. 

•  Standardized  -  They  are  enabling  interoperability  and  reuse  of  individual  modules  and  other  learning 
content. 

•  Easy  to  Use  -  They  are  becoming  more  user  friendly,  especially  as  instructional  tasks  become 
increasingly  complex. 

The  need  for  ultra-agile  authoring  that  can  be  done  rapidly  on  site  by  Subject  Matter  Experts  (SMEs)  continues 
to  increase  in  importance.  To  a  large  extent,  this  need  is  driven  by  the  complex,  volatile  demands  of  today’s 
coalition,  asymmetric  operations.  It  is  also  driven  by  the  Web  2.0  culture,  expectations,  and  work  habits  of 
today’s  participants,  who  are  used  to  constant  peer-to-peer  communication  via  capabilities  such  as  email, 
texting,  blogs,  wikis,  podcasting,  Twitter,  on-line  games,  folksonomies,  YouTube,  and  whatever  comes  next. 

The  key  to  enhancing  capability  in  these  efforts,  as  in  all  authoring  efforts,  is  to  deal  with  the  trade-off 
between  flexibility  in  application  and  simplicity  of  use  -  a  trade-off  that  inevitably  occurs  in  all  human- 
computer  interactions.  As  suggested  by  Figure  6-1,  it  is  possible  to  create  very  simple  authoring  languages  by 
limiting  what  one  can  do  with  them  -  reducing  flexibility,  increasing  simplicity.  It  is  also  possible  to  use 
authoring  languages  to  implement  quite  elaborate  and  clever  instructional  designs  but  only  at  the  expense  of 
simplicity.  Progress  in  authoring  language  development  and  design  can  be  seen  in  advanced  technical  ideas 
for  pushing  back  this  trade-off  and  increasing  flexibility  while  decreasing  or  at  least  maintaining  simplicity  of 
use.  Achieving  more  flexibility  or  more  simplicity  alone  is  seldom  enough.  Advances  in  authoring  are  keyed 
to  discovering  and  developing  ways  to  do  both  in  the  same  system. 
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Figure  6-1:  Authoring  Trade-Off  Between  Simplicity  and  Flexibility. 

The  Task  Group  found  examples  of  advanced  authoring  in  four  areas: 

1)  Development,  modification,  and  maintenance  of  simulations  in  the  field; 

2)  Development,  modification,  and  maintenance  of  simulations  for  operating  and  maintaining  equipment; 

3)  Semi-automated  generation  of  instructional  material  from  standard  office  products  such  as  word 
processing  documents  and  projection  slides;  and 

4)  Peer  production  arising  from  Web  2.0  capabilities. 

A  perennial  characteristic  of  military  operations  is  surprise.  Nothing  is  more  certain  than  the  unexpected. 

Many  aspects  of  any  operation  can  be  anticipated,  planned,  and  prepared  for.  However,  with  every  action 
taken,  countermeasures  will  be  initiated,  thereby  precipitating  a  need  for  counter-countermeasures  while  the 
entire  operation  ratchets  up  into  the  usual  chaos  of  confusion.  Added  to  these  complications  are  the  cultural 
aspects  of  today’s  asymmetric  operations  and  the  need  for  coalition  cooperation.  Lessons  learned  in  the  field 
need  to  be  rapidly  and  widely  disseminated  -  in  a  matter  of  hours  if  possible. 

To  some  extent  we  can  prepare  individuals  to  recognize  and  deal  with  unexpected  events  through  training 
[32].  Particularly  effective  is  training  that  prepares  individuals  by  placing  them  in  simulated  environments  that 
expose  them  to  the  unexpected  events  that  are  likely  to  occur  in  theater.  Few  people  are  likely  to  know  and 
understand  these  events  as  well  as  those  people  who  are  actually  in  the  field  of  operations.  A  capability  that 
enables  them  to  rapidly  (i.e.,  in  a  matter  of  hours)  author  and/or  edit  simulations  that  incorporate  newly 
experienced  events  and  distribute  them  to  others  in  the  field  will  contribute  significantly  to  attaining  the 
resilience  needed  to  deal  with  these  events.  With  this  possibility  in  mind,  DARPA  has  undertaken  development 
of  authoring  capabilities  to  “eliminate  middlemen”  (i.e.,  software  programmers)  and  allow  end  users  to  create  or 
edit  their  own  simulations.  As  described  by  Kaufman  and  Lunceford  [33]: 
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The  overarching  goal  of  the  RealWorld  program  is  [by  using  a  commercial  game  engine]  to  enable  non¬ 
programmers  to  rapidly  author  highly  accurate,  geo-specific  virtual  worlds  and  simulations.  RealWorld 
strives  to  put  the  power  of  “virtual  reality”  directly  into  the  hands  of  the  people  who  do  the  work,  are  the 
experts,  and  often  have  a  life-risking  stake  in  the  outcome.  Its  approach  is  to  provide  a  robust  set  of 
easy-to-use  scenario  development/authoring  tools  that  users,  after  very  little  instruction,  can  use  to  create 
state-of-the  art  3D  simulations  for  training,  pre-mission  practice,  analytical  evaluation  of  possible  future 
events,  and  post-mission  lessons-leamed  analysis.  RealWorld  accomplishes  these  ends  while  also 
driving  down  the  cost  and  time  lines  of  simulation  development.  Its  tools  can  be  placed  in  the  hands  of 
every  person  within  an  organization  who  has  a  stake  in  the  successful  execution  of  its  missions  (p.  1). 

RealWorld  is  particularly  valuable  for  time-critical  applications  where  the  ability  to  train  and  prepare 
individuals  and  teams  in  a  highly  accurate  and  realistic  simulation  can  save  lives,  including  those  of 
non-combatants,  and  reduce  mission  failure  risk.  No  matter  how  short  the  mission  timeline,  “practice 
makes  perfect”  applies  and  a  simulation  system  that  cannot  be  quickly  adapted  to  local  needs  is  of 
little  use  (p.  1). 

RealWorld  provides  capabilities  for: 

•  Terrain  Editing 

•  Ability  to  place  objects  from  vector  files  (e.g.,  rivers  and  roads)  in  their  correct  locations; 

•  Ability  to  adjust  terrain  elevations  (e.g.,  properly  place  a  road  cut  into  a  mountain  side);  and 

•  Ability  to  specify  ground  textures  (e.g.,  grass,  sand,  gravel,  paving). 

•  Creating  Building  Interiors  and  Exteriors 

•  Ability  to  import  a  wide  range  of  structures  from  existing  sources  (e.g.,  OpenFlight  and  Collada 
files); 

•  Ability  to  automatically  texture  and/or  adjust  texturing  of  objects  in  shape  files  obtained  from 
standard  sources  (e.g.,  Laser  Intensity  Direction  And  Ranging  (LiDAR)  files); 

•  Ability  to  import  building  interiors  from  Computer-Aided  Design  (CAD)  files;  and 

•  Ability  to  generate  geo-specific  building  interiors  and  exteriors  by  employing  an  easy-to-use  tool. 

All  of  these  capabilities  are  available  through  simple  drag  and  drop  functions  involving  an  object  database  that 
will  continue  to  grow  in  size  and  diversity. 

Figure  6-2  shows  scenes  from  a  simulation  covering  about  1  square  mile  and  including  32  geo-typical  and  geo¬ 
specific  buildings,  435  objects  such  as  cars  and  trees,  a  checkpoint,  and  a  hidden  cache  of  weapons.  It  was 
authored  in  one  day  by  an  individual  who  was  neither  a  software  programmer  nor  an  experienced  simulationist. 
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Figure  6-2:  Contraband  Search  -  An  Example  Covering  About  1  Square  Mile. 


RealWorld  authoring  also  includes  a  3D  audio  system  that  allows: 

•  Simulation  participants  to  communicate  with  each  other  if  they  are  within  earshot  range  and  via 
simulated  radio  (multi-channel  with  frequency  hopping,  encryption,  realistic  jamming  vulnerabilities, 
and  so  forth)  if  they  are  not;  and 

•  Adjustment  of  the  type  and  location  of  sound  (e.g.,  an  explosion)  as  well  as  its  intensity  based  on  its 
distance. 

Finally,  RealWorld  includes  a  realistic  electronic  warfare/threat  environment  to  allow  for  integrated  mission 
planning  and  rehearsal.  This  environment  can  be  used  in  stand-alone  mode,  or  it  can  be  integrated  with  the 
ground  environment  and  radio  system.  It  enhances  the  immersion  of  simulation  participants  by  adding 
functionalities  beyond  the  visual  range  and  imagery.  Figure  6-3  shows  an  example  of  the  RealWorld  electronic 
warfare,  threat,  and  communications  enviromnent. 
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Figure  6-3:  The  RealWorld  Electronic  Warfare,  Threat,  and  Communications  Environment. 

A  particular  strength  of  RealWorld  is  its  ability  to  link  mission  planning  to  mission  rehearsal,  both  of  which 
tend  to  be  developed  independently  using  duplicative  databases.  RealWorld  automatically  links  planning  data 
such  as  mission  tracks,  obstacles,  cultural  annotations,  and  the  like  to  display  the  same  data  in  both  mission 
planning  and  mission  rehearsal  applications.  This  capability  allows  planners  to  manipulate  data  (i.e.,  author) 
in  either  database,  working  in  either  two  dimensions  or  three  dimensions,  whichever  provides  the  best  view  of 
the  mission.  RealWorld  also  provides  tools  to  determine  line  of  sight  and  path  across  the  terrain. 

It  has  been  long  noted  [5]  that  both  training  and  performance  aiding  can  be  generated  from  single  underlying 
knowledge  base,  which  in  the  case  of  a  device  could  be  a  model  representing  its  components  and  their 
interactions.  An  early  instance  of  training  and  performance  aiding  combined  in  this  manner  was  the  SOPHIE 
system  [34],  SOPHIE  used  a  general-purpose  electronic  simulator,  SPICE  [35],  equipped  with  a  mathematical 
model  of  a  regulated  power  supply,  to  create  a  reactive,  tutorial  environment  for  training  in  electronic 
troubleshooting. 

The  underlying  model  allowed  single  or  multiple  component  faults  to  be  introduced  either  by  a  (human) 
instructor  or  by  SOPHIE  itself  and  be  propagated  throughout  the  (simulated)  power  supply  in  a  comprehensive 


RTO-TR-HFM-129 


6-5 


FUTURE  OF  AUTHORING 


ORGANIZATION 


and  vertical  fashion.  SOPHIE  employed  a  semantic  network  of  information  about  the  power  supply  (one  of  the 
most  sophisticated  natural  language  processors  ever  built  into  ITS)  and  an  articulate,  expert  troubleshooting 
capability  that  could  not  only  locate  and  identify  arbitrary  faults,  but  also  guide  students  in  solving  them. 

SOPHIE  responded  in  a  realistic  fashion  to  the  actions  and  decisions  of  a  student  engaged  in  electronic 
troubleshooting,  either  for  performance  aiding  or  for  training.  It  employed  strategies  that  shadowed  those  used 
by  the  student  to  make  inferences  about  the  circuit  based  on  the  student’s  measurements  and  its  own  internal 
mathematical  model  of  the  circuit.  SOPHIE  could  thereby  determine  which  hypotheses  the  student  or  problem 
solver  was  testing,  construct  a  model  of  his/her  state  of  knowledge,  and  use  this  model  to  generate  tailored 
explanations  that  the  specific  student/user  would  be  prepared  to  understand.  Its  “articulate  expertise” 
pioneered  capabilities  that  are  still  lacking  in  many  of  today’s  information  acquisition  and  retrieval  systems. 

However,  even  with  its  pre-supplied  model,  authoring  for  SOPHIE  was  expensive  and  time-consuming  and 
required  the  services  of  highly  experienced  Lisp  programmers.  ReAct  and  DIAG  are  examples  of  serious  efforts 
to  retain  SOPHIE-like  “intelligent”  functionalities  while  significantly  reducing  both  the  time  and  cost  to  produce 
systems  of  this  sort  [36],  [37],  [38].  These  efforts  follow  a  model  of  authoring  software  resources  illustrated  in 
Figure  6-4,  which  range  from  general-purpose  programming  languages  (e.g.,  C++,  Java),  to  authoring  languages 
for  interactive  graphics  and  animations  (e.g.,  Flash,  Vcom  3D),  to  authoring  languages  for  model  development 
(e.g.,  ReAct  and  Vcom3D),  to  application-specific  languages  (e.g.,  Diagnostic  Discrete  Decision  Making 
(D3M),  DIAG,  and  Personal  Knowledge  Source  (PKS)). 


D3M,  DIAG,  and  PKS  are  all  based  on  models  built  by  ReAct,  which,  in  turn,  requires  a  mid-level  programmer 
who  has  knowledge  of  Flash.  DIAG  requires  rudimentary  knowledge  of  the  ReAct  model  structure  and 
development  but  not  a  deeper  knowledge  of  programming.  Development  for  operating  and/or  maintaining  a 
specific  or  generic  device,  then,  begins  with  the  development  of  a  ReAct  model  of  the  device  by,  perhaps  a 
mid-level  programmer,  followed  by  DIAG  development  of  diagnostic  practice  or  troubleshooting  by  a  designer 
who  has  a  thorough  knowledge  of  the  device  (specific  or  generic)  but  has  no  particular  expertise  in  computer 
programming. 
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The  requirement  for  the  ReAct  model  developer  is  not  as  formidable  as  it  may  seem  because  ReAct  expands, 
extends,  and  simplifies  Flash  capabilities.  Among  other  things,  it  provides: 

•  Libraries  of  object  prototypes  for  building  models  that  include  both  the  graphic  representations  and 
behavior  of  the  objects.  For  example,  an  arrow  for  a  dial  object  may  “know”  that  it  only  rotates 
clockwise  or  counter-clockwise  in  two  dimensions  and  is  connected  to  the  dial.  Beyond  this,  ReAct 
objects  may  be  physical  (e.g.,  the  above  arrow),  functional  (e.g.,  schematic  elements),  or  symbolic 
(e.g.,  a  radar  blip  representing  the  intentions  and  capabilities  of  specific  aircraft). 

•  A  built-in  simulation  engine  that  responds  to  various  inputs,  including  those  from  the  user  and  the 
passage  of  time. 

•  Real-time  functions  for  producing  controlled  animations  that  are  integrated  with  the  simulation 
engine. 

•  An  object-oriented  architecture  that  allows  ReAct  objects  to  operate  independently,  exchange  data 
with  each  other,  and  combine  with  other  objects  in  the  fashion  recommended  by  Boot’s  [39]  building 
block  solutions  for  authoring  instructional  software.  Each  object  contains  information  about  itself  and 
about  the  objects  with  which  it  can  interact,  including  how  it  must  interact  with  them. 

Even  though  ReAct  requires  some  knowledge  of  the  Flash  programming  that  it  expands,  extends,  and  simplifies, 
it  drastically  reduces  the  time,  effort,  and  cost  of  model  development.  For  instance,  consider  the  two  dials  shown 
in  Figure  6-5.  Both  dials  have  identical  functionality.  Both  objects  rotate  as  a  mouse  is  dragged  about  their 
centers,  snap  into  the  nearest  detent  when  the  mouse  is  released,  and  notify  affected  objects  of  their  state. 
Flowever,  the  dial  on  the  left  took  86  program  statements  to  create  and  the  dial  on  the  right  took  one  ReAct 
statement  to  create. 


A  Rotary  Dial  with  Detents 


Flash  Object 
86  program  statements 

AECM_ON 


OFF 


AECM_Stndby 


ReAct  Object 
1  program  statement 

AECM  ON. 


OFF 


AECM_Stndby 


Figure  6-5:  Rotary  Dials  Created  with  Flash  (Left)  and  ReAct  (Right). 
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DIAG  was  specifically  developed  to  operate  upon  ReAct  models  in  providing  guided  maintenance  practice 
and  diagnostic  performance  support.  It  allows  users  working  with  the  models  to  insert  and  explore  known 
faults,  insert  and  diagnose  unknown  faults,  get  help  reviewing  symptoms,  get  help  picking  next  system  test, 
and  get  help  in  making  inferences.  The  primary  DIAG  functions  involve  problem  selection,  recording  and 
interpreting  learner  (or  user)  performance,  providing  assistance,  and  debriefing  the  leamer/user  following 
his/her  problem  solution.  To  implement  training,  the  author  simply  has  to  hand  the  faults(s)  to  DIAG,  which 
then  invokes  the  ReAct  simulation  engine  to  set  the  indicators  and  test  points. 

DIAG  is  assisted  in  this  process  by  fuzzy  fault  effect  knowledge,  which  is  based  on  fuzzy  set  theory 
(e.g.,  [40]).  Here,  the  notion  is  that  a  particular  symptom  may  not  always  occur.  In  the  case  of  DIAG  assessing 
a  ReAct  model  with  a  given  fault,  seven  possibilities  for  an  indicator  or  symptom  -  ranging  from  always 
occurring  to  never  occurring  -  are  allowed.  Such  “fuzzy”  possibilities  are  particularly  important  in  learning 
troubleshooting  for  multi-component  failures  and  intermittent  component  failures. 

The  efficiencies  permitted  by  DIAG  in  authoring  for  equipment  maintenance  and  operation  training  or 
performance  aiding  were  reported  by  Towne  [37].  The  training  to  maintain  a  shipboard  radar  transmitter 
with  86  replaceable  units,  17  screens,  and  31  controls  took  4  domain-expert  person-weeks  and  5  authoring 
person-weeks  to  develop.  An  aircraft  power  distribution  system  with  148  replaceable  units,  28  screens, 
and  10  controls  took  1  domain-expert  person- week  and  3  authoring  person-weeks  to  develop. 

DAIG  and  ReAct  have  been  available  for  wider  use  for  about  5  years.  Given  the  ubiquity  of  requirements  for 
equipment  maintenance  and  operation  and  the  efficiencies  to  be  gained  from  use  of  ReAct  and  DIAG,  it  is 
puzzling  not  to  find  them  more  widely  used. 

One  way  to  provide  rapid,  agile,  inexpensive  authoring  is  to  start  with  standard  materials  such  as  Word 
documents  and  PowerPoint  slides,  with  which  millions  of  non-programmers  are  familiar.  The  computer  then 
does  the  work  of  extracting  their  content  and  shaping  it  into  interactive  training  and  performance  aiding 
materials  that  can  be  delivered  by  computer  technology. 

Several  authoring  tools  are  emerging  that  make  this  approach  possible.  They  take  word  processed  documents 
or  projection  slides  developed  by  a  (non-programming)  producer  as  input  and,  using  either  hypertext  markup 
or  Graphical  User  Interface  (GUI)  procedures,  help  the  producer  structure  and  transform  this  input  into  a 
database  from  which  computers  can  generate  interactive,  adaptive  instruction  or  performance  aiding.  Using 
the  word  processor’s  or  slide  producer’s  font  type  and  style  conventions,  assigning  a  hierarchical  structure 
using  different  level  headings  to  indicate  ancestor  and  descendent  sections,  and  assigning  names  to  the 
headings,  the  tool  produces  a  markup  representation,  which  translates  directly  into  a  computer  program  or  at 
least  a  file  of  material  with  commands  for  its  computer  delivery. 

The  first  systems  of  this  sort  used  various  Hypertext  Markup  Language  (HTML)  conventions  for  this  sort  of 
authoring,  which  often  handicapped  portability  of  the  resulting  instructional  material.  As  Brusilovsky  [41]  points 
out,  this  problem  went  away  with  the  advent  of  Extended  Markup  Language  (XML),  which  not  only  specifies 
markup  conventions,  but  has  also  attracted  the  development  of  a  large  number  of  tools,  editors,  and  parsers  that 
conform  to  these  conventions.  Because  markup  alone  offers  considerable  freedom,  it  also  provides  numerous 
opportunities  to  make  subtle  errors  that  are  hard  to  find.  In  addition,  although  Microsoft  Word  can,  of  course, 
create  HTML  documents  by  itself,  the  code  it  creates  cannot  be  easily  edited  by  available  tools,  and  it  is  difficult 
to  make  minor  curriculum  modifications  in  Word-generated  HTML  documents.  Again,  as  Brusilovsky  notes, 
a  variety  of  checking  tools  have  to  be  provided  to  help  non-programming  authors  find  errors  and/or  make 
modifications. 
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Most  systems  designed  to  assist  non-programmers  in  developing  adaptive  materials  from  ordinary  documents 
and  slides  use  a  GUI,  with  pop-up  menus,  drag  and  drop  facilities,  concept  maps,  and  the  like  to  guide  users 
through  the  use  of  form-based  tools.  Many  of  these  systems  provide  an  input  form  for  entering  information 
about  a  concept  and  another  form  for  specifying  its  display. 

HTML  and  XML  documents  are  page  oriented.  They  expect  to  display  a  page  of  material  (which  may  allow 
scrolling,  so  it  is  not  limited  to  a  specific  page  size)  and  wait  for  something  to  happen.  It  is  easy  to  use  these 
semi-automated  systems  to  generate  material  that  marches  through  instruction  in  a  linear  fashion  in  response 
to  a  learner  typing  a  “Next”  key.  Simple  materials  of  this  sort  can  be  produced  quickly  and  inexpensively, 
but  materials  with  rich  content  and  highly  adaptive  information  structures  tend  to  be  much  more  complicated 
and  expensive  to  develop  -  to  the  point  where  it  might  be  better  to  hire  a  developer  who  at  least  has  sufficient 
programming  skills  to  use  an  authoring  or  scripting  language. 

This  is  not  to  argue  against  the  production  of  very  simple  materials.  Large  amounts  of  adaptive  hypermedia 
materials  can  be  produced  from  existing  content  in  a  matter  of  hours  and  made  available  globally  on  the  web 
or  whatever  the  global  information  infrastructure  becomes  in  the  future.  However,  it  may  be  possible  to 
augment  these  materials  with  more  sophisticated  tutoring  capabilities.  The  capabilities  developed  for  the 
REDEEM  authoring  environment  were  designed  to  allow  classroom  teachers,  using  a  simple  GUI,  to  adapt 
relatively  linear,  non-adaptive  computer-assisted  learning  materials  to  their  own  presentation  and  delivery 
methods.  Tests  have  shown  that  applying  REDEEM  to  existing  course  content  took  2-25  hours  to  develop  an 
hour  of  newly  adaptable  instruction  that  could  produce  significant  improvements  in  learning  with  an  average 
effect  size  of  0.76  [42].  REDEEM,  in  combination  with  the  semi-automated  capabilities  described  here,  might 
be  significantly  cost  effective  in  comparison  with  other  authoring  techniques. 

Gradually,  the  capabilities  of  authoring  systems  that  allow  non-programming  domain  specialists  to  produce 
highly  effective  instructional  materials  through  guided  conversations  with  these  systems  are  evolving.  We  are 
not  yet  able  to  generate  instructional  interactions  entirely  by  computer,  but  we  seem  to  be  heading  toward  a 
future  in  which  domain  knowledge  will  be  pulled  out  of  the  global  information  infrastructure  -  on  demand  and 
in  real  time  -  to  meet  the  specific,  individual  needs  of  a  learner  or  problem  solver  on  an  interaction  to 
interaction  basis.  In  the  meantime,  we  will  have  to  choose  between  a  plethora  of  authoring  capabilities. 


6.1  PEER  PRODUCTION  (SEE  [43] ) 

Peer  production  is  an  emerging  authoring  capability  that  complements  other  production  resources.  The  rapidly 
increasing  range  and  extent  of  human  networks,  made  possible  by  computer-mediated  information  and 
communication,  enable  a  qualitative  difference  in  capability.  It  is  a  “more  is  different”  type  of  change. 

Example  Scenario  -  Corporal  Jones  had  seen  it  happen.  He  was  the  driver  of  the  rear  vehicle  when  the 
explosion  occurred.  This  time,  there  had  been  no  casualties.  He  was  both  relieved  and  pleased,  knowing  that  he 
had  personally  been  involved  in  this  success.  However,  this  was  not  the  case  earlier  when  a  similar  roadside 
bomb  had  caused  one  death  and  several  wounded.  Three  weeks  ago,  Corporal  Jones  had  been  driving  just 
behind  the  lead  vehicle  and  had  seen  events  unfold  just  prior  to  the  explosion.  Immediately  following  that, 
he  had  spoken  his  recollections  into  his  Personal  Digital  Assistant  (PDA),  taking  care  to  describe  the  events 
prior  to  the  explosion.  He  also  used  the  PDA ’s  camera  function  to  take  pictures  of  the  scene.  Inside  the 
compound  that  night,  he  updated  his  blog  and  downloaded  the  pictures.  He  carefully  tagged  his  entries,  knowing 
that  they  could  be  used  by  others  to  learn  about  the  incident.  Over  the  next  couple  of  days,  many  members 
accessed  his  blog  and  pictures.  The  rapid-training  team  in  theater  used  Jones  ’  account  and  the  accounts  of 
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others  to  update  the  “Ambush  simulation  game”  used  to  train  drivers  in  theatre.  This  update  was  also  instantly 
available  on  the  network  and  was  immediately  used  to  train  the  next  rotation.  Jones  ’  account  was  voted  by  his 
peers  as  “highly  relevant  and  useful”  and  now  appeared  on  the  top  of  the  wiki-based  lessons-learned  flash  page. 
So,  when  the  lead  driver  recognized  the  potential  danger  and  took  evasive  action,  no  luck  was  involved.  The  vast 
network  and  its  social  production  tools  ensured  that  Jones  ’  knowledge  was  quickly  and  effectively  distributed. 

6.1.1  Commons-Based  Peer  Production 

Benkler  [44]  identifies  “commons-based  peer  production”  as  a  form  of  human  collaboration.  He  defines  it  as 
decentralized  information  gathering  that  depends  on  large  aggregations  of  individuals  who  self-identify  for 
tasks  and  independently  search  their  information  environment.  The  fundamental  advantage  of  commons-based 
peer  production  is  that  it  identifies  and  marshals  human  creativity  and  knowledge  for  work  on  information  and 
cultural  resources. 

6.1.2  Peer-to-Peer  Learning 

As  the  example  scenario  suggests,  network  capabilities,  continuous  learning,  and  effective  operational  agility 
are  being  interwoven  within  a  new  culture  and  operational  paradigm.  In  the  future,  each  individual  may  build 
his/her  own  learning,  working,  and  operational  environment.  All  the  required  information  and  collaborative 
tools  and  services  will  be  integrated  into  a  single  interface  -  akin  to  today’s  web  browser.  Especially 
important  may  be  the  way  in  which  knowledge  and  information  are  used  and  shared.  The  days  of  the  senior 
staff  as  the  only  valid  source  of  knowledge  may  be  numbered.  The  emerging  peer-to-peer  model  fits  in  well 
with  today’s  complex  and  volatile  battlespaces  where  the  knowledge  and  intelligence  of  the  collective  are 
essential  for  victory. 

While  we  seem  to  be  making  progress  toward  this  model,  military  forces  remain  structured  to  support 
traditional  modes  of  transmission,  with  platform  instructors  in  front  of  the  class.  Current  knowledge-sharing 
tools  are  limited.  Compliance  training  will  always  be  required,  but  the  increasing  influence  of  peer  production 
seems  inevitable.  Our  young  students,  based  on  their  experience  in  the  Middle  East,  may  have  much  more 
current  operational  experience  than  their  trainers.  Failure  to  evolve  from  the  current  top-down  curriculum 
design  model  to  one  that  is  more  socially  developmental  may  leave  us  trapped  in  “training  for  the  last  war”. 

6.1.3  Peer  Production  as  a  Human  Capability  Multiplier 

Peer  production  promises  to: 

•  Integrate  continuous  learning  and  enhance  operational  agility; 

•  Reduce  the  coordination  costs  of  time,  effort,  resources,  and  people; 

•  Increase  the  pool  of  available  skills,  knowledge,  and  judgment  that  can  be  brought  to  bear  on  operations; 

•  Increase  accountability,  responsibility,  and  responsiveness;  and 

•  Allow  the  organization  to  marshal  more  of  its  human  capability /capital  for  productive  and  operational 
ends. 

In  effect,  peer  production  can  act  as  a  human  capability  multiplier  by  allowing  the  people  involved  in 
administration  and  coordination  to  be  redeployed  into  operations.  It  can  also  reduce  the  cost  and  negative 
effects  of  the  replacement  cycle  by  increasing  continuity  and  corporate  memory,  thereby  powering  functional 
responsibilities  with  wider  virtual  divisions  of  labor. 
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6.1.4  Peer  Production  in  a  Military  Context 

Peer  production  in  a  military  context  can  build  a  common  picture  (situation  awareness),  cooperatively  solve 
problems  (tactics),  and  maintain  a  progressive  discourse  (continual  improvement  and  sense-making). 
Operational  environments  have  far  too  much  data,  information,  and  knowledge  for  any  single  person  to 
comprehend.  The  “work  of  the  masses  -  wisdom  of  the  crowd”  will  be  the  only  way  to  make  sense  of  it  all. 
As  Corporal  Jones  shows,  information  and  sharing  of  experiences  must  feed  back  into  all  layers  of  military 
organizations. 

In  addition  to  knowledge  sharing,  peer  production  enhances  relationships  between  the  members  of  the 
community,  which,  in  turn,  enables  the  members  to  determine  how  and  where  they  can  further  serve  other 
members.  It  increases  the  organization’s  “social  capital.”  Relationships,  trust,  and  a  sense  of  professional 
community  are  critical  factors  for  effective  connections,  communication,  and  problem  solving.  They  set 
the  stage  for  current  and  future  operational  agility  and  increase  the  development  of  operational  cohesion. 
Peer  production  enables  a  connected,  decentralized  network  of  individuals  to  access  knowledge  and  resources 
that  might  otherwise  be  unavailable. 

6.1.5  Peer  Production  Technologies 

The  capabilities  of  Web  2.0  have  only  recently  emerged  (e.g.,  [45],  [46],  [47]).  Three  of  them  seem  particularly 
relevant  to  peer  production: 

1)  Wikis  -  A  wiki  is  a  website  with  specialized  software  that  allows  its  visitors  to  easily  add,  remove, 
and  otherwise  edit  and  change  its  content.  This  ease  of  interaction  and  operation  makes  a  wiki  an 
effective  tool  for  peer  production.  Generally,  no  review  takes  place  before  modifications  are  accepted, 
and  most  wikis  are  open  to  the  general  public,  without  the  need  to  register  a  user  account.  However, 
participants  may  be  requested  to  log-in  to  acquire  a  “wiki-signature”  for  auto-signing  edits, 
and  private  wiki  servers  require  user  authentication  to  edit  -  and  sometimes  even  to  read  -  pages. 

2)  Blogs  -  If  wikis  are  sites  of  collective  participation,  blogs  are  individual  productions.  A  blog  can  best 
be  thought  of  as  an  online  personal  journal  that  can  embody  text,  audio,  and  video  dimensions.  Blogs 
can  be  used  to  capture  the  constant  chatter  of  the  collective  intelligence.  Individuals  can  use 
aggregators,  such  as  Bloglines,  in  conjunction  with  Really  Simple  Syndication  (RSS)  to  automatically 
receive  information  that  is  important  to  them.  The  information  contained  within  blogs  can  also  be 
referenced,  shared,  and  transferred  into  a  lessons-leamed  wiki  to  improve  training  and  operational 
tactics,  as  may  have  occurred  with  Corporal  Jones’  information. 

3)  RSS  and  Aggregators  -  RSS,  or  Really  Simple  Syndication,  is  one  of  the  gems  of  Web  2.0 
technologies.  It  allows  users  to  “subscribe”  to  any  piece  of  changing  information  on  the  net.  Content 
can  come  from  media  sites,  blogs,  workgroups,  and  so  forth  as  long  as  they  have  been  set  up  for  RSS. 
This  technology  allows  one  to  keep  track  of  large  amounts  of  rapidly  emerging  information  easily  and 
concisely.  It  enables  the  mass  of  information  likely  to  be  produced  by  peer  production  to  be  managed 
effectively  and  makes  peer  production  a  practicable  resource  for  training  and  performance  aiding. 
Again,  as  may  have  occurred  with  Corporal  Jones,  RSS  allowed  others  to  be  alerted  without  extra 
effort  on  their  part. 

All  of  these  capabilities  can  be  easily  acquired  and  installed  using  open-source  software. 
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6.1.6  Peer  Production  Trials:  Lessons  Learned 

A  small  group  of  early  adaptors  in  the  Canadian  Forces  set  out  to  explore  wikis,  blogs,  and  RSS.  This  process 
is  continuing,  even  after  the  “launch”  of  a  wiki  and  blogs.  Some  of  the  lessons  learned  follow: 

•  Getting  the  Software  on  the  Network 

•  Lesson  Learned  -  Running  software  is  imperative  before  one  is  able  to  sell  the  idea  to  others. 
Many  free  wikis  (e.g.,  http://pbwiki.com/)  can  be  used  while  paperwork  is  being  processed. 
Use  care  in  posting  information  discussions  in  the  open  arena  of  the  Internet. 

•  Convincing  Others  the  Way  of  the  Wiki  -  Inevitably,  people  have  a  difficult  time  with  the  concept 
of  the  wiki  as  being  able  to  be  edited  by  all.  Much  discussion  (http://en.wikipedia.org/wiki/Wiki; 
[48])  of  that  issue  and  those  related  to  it  has  ensued.  Most  wikis  allow  for  easy  restoration  of  previous 
content,  provide  a  complete  history  of  edits  (including  who  made  the  edit),  and  have  a  designated 
space  for  discussion  associated  with  the  article. 

•  Lesson  Learned  -  Ensure  that  there  are  no  anonymous  users  on  your  wiki.  Have  them  use  their 

official  work  email  address  and  real  names.  This  caution  eliminates  vandalism  and  misuse. 

•  Canadian  Forces  Wiki  -  The  Canadian  Forces  wiki  was  never  publicized.  In  6  months,  membership 
was  well  over  150  users  and  over  100  articles.  Users  have  been  free  to  use  the  wiki  for  whatever 
purpose  they  deem  fit.  Very  little  support  was  provided,  but,  by  all  measures,  the  wiki  is  being  used 
successfully.  Most  users  just  open  an  article  for  editing  and  pick  up  the  wiki-coding  by  themselves. 
If  peer  production  is  to  take  up  some  of  the  authoring  burden,  the  costs  of  doing  so  will  be  minimal. 

•  Lesson  Learned  -  Find  a  group  that  is  already  doing  something  for  which  a  wiki  could  improve 

a  person’s  ability  to  do  it.  Start  small  and  let  it  grow  organically.  Do  not  force  people  to  use  the 
wiki.  Do  not  limit  access.  Do  not  restrict  reading  or  editing  rights. 

•  Canadian  Forces  Blogs  -  Blogs  were  also  introduced  to  the  Canadian  Forces  with,  again,  very  little 
advertisement.  Further,  less  than  10%  of  Canadian  Forces  have  access  to  these  blogs.  Despite  these 
limitations,  a  regular  group  of  bloggers  is  sharing  information.  Reading  and  commenting  on  others’ 
blogs  by  the  core  group  seems  to  be  keys  to  building  trust  and  increasing  use  of  the  blogs.  Ultimately, 
the  increased  use  of  blogs  will  require  open  access  and  expanded  scope.  However,  the  Canadian 
Force  is  a  “need-to-know”  organization  whereas  peer  production  promotes  a  “need-to-share”  culture. 
Policies  and  techniques  will  have  to  be  developed  before  peer  production  can  become  widely 
implemented. 

•  Lesson  Learned  -  Allow  users  to  explore  how  blogs  can  be  useful  to  them.  Encourage  use  by 
reading  and  commenting  on  others’  blogs. 

•  Canadian  Forces’  Use  of  RSS  and  Aggregators  -  Canadian  Forces’  use  of  RSS  and  aggregators  has 
been  limited  to  3  -  4  early  adaptors.  These  technologies  have  been  introduced  at  several  conferences 
and  have  engendered  considerable  interest.  The  Canadian  Forces  have  many  readily  identifiable 
groups  that  could  easily  benefit  from  the  ability  to  share  their  “favorites.”  RSS  and  aggregators  help 
ensure  receipt  of  the  most  current  information  delivered  anytime,  anywhere.  However,  the  Canadian 
Forces  are  not  able  to  use  aggregators  within  their  firewall.  A  separate  computer  and  network  must  be 
used.  Again,  policies  and  techniques  must  be  developed  before  the  promise  of  these  capabilities  can 
be  fully  realized. 

•  Lesson  Learned  -  With  limited  resources,  promote  the  peer  production  technologies  that  will 
have  the  greatest  effect  on  the  organization  and  can  be  implemented  with  minimal  impact  on 
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current  policies.  The  technology  must  be  used  by  users  to  gain  insights  into  its  functionality  and 
usefulness. 

6.1.7  Peer  Production  Challenges 

Peer  production  holds  considerable  promise  for  easing  authoring  costs,  but  it  is  still  in  the  very  early  stages 
of  adoption.  To  reach  the  state  of  sharing  and  learning  that  we  saw  in  the  Corporal  Jones  scenario, 
some  challenges  must  be  met.  Dealing  with  a  “need-to-know”  organization  that  needs  to  develop  the  trust  and 
access  required  by  a  “need-to-share”  technology  will  take  time.  However,  its  value  is  likely  to  be  understood 
and  perhaps  demonstrated  by  the  new  generation  of  people  in  the  Canadian  Forces. 

A  slow  revolution  appears  to  be  changing  our  views  of  knowledge.  We  are  moving  away  from  objectivism 
and  toward  a  social  constructivism  where  reality  is  a  collective  interpretation.  Critical  thinking  will  be  especially 
important  in  justifying  assertions  and  judging  the  validity  of  data,  information,  and  knowledge,  regardless  of  the 
source.  It  may  be  at  the  root  of  peer  production’s  success  of  failure.  Nonetheless,  the  trials  thus  far  have  been 
encouraging. 

Other  developments,  such  as  RealWorld  authoring,  are  in  accord  with  the  authoring  environment  evolving 
from  Web  2.0  capabilities  in  which  everyone  is  an  author,  including  the  learners  and  users  themselves. 
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Publications  on  and  discussions  of  how  to  choose  authoring  products  abound  (e.g.,  [49],  [50]).  The  Task  Group 
decided  early  on  to  mention  specific  products  as  examples  only  and  to  avoid  specifically  recommending  any  for 
at  least  three  reasons: 

1)  The  product  list  is  constantly  changing. 

2)  The  recommendation  of  any  one  or  group  of  capabilities  creates  issues  with  those  not  selected  or 
inadvertently  neglected. 

3)  Primarily,  however,  recommending  any  specific  capabilities  to  a  general  audience  is  impossible. 
The  choice  depends  on  who  will  use  them  to  do  what  and  under  what  constraints  of  cost,  time,  and 
other  resources.  Few  one-size-fits-all  authoring  capabilities  exist. 

With  these  considerations  in  mind  and  after  reviewing  a  variety  of  the  many  guidelines  for  selecting  specific 
authoring  capabilities  and  tools,  the  Task  Group  arrived  at  26  summary  issues.  These  issues  are  listed  in  Table 
7-1  and  discussed  in  the  paragraphs  following  the  table.  These  issues  suggest  matters  to  be  considered  by 
anyone  selecting  authoring  capabilities  for  computer-mediated  instruction.  They  are  based  on  an  early  paper 
prepared  for  the  Task  Group  by  Boot  and  Kitchen  [51]. 


Table  7-1:  Issues  to  Consider  in  Selecting  Authoring  Capabilities  and  Tools. 


What  does  the  tool  do? 

Who  will  be  developing  the  instruction? 

What  is  the  quality  of  the  tool? 

What  does  the  tool  cost? 

How  reliable  is  the  supplier  providing  the  tool? 

How  widely  used  is  the  tool? 

What  technical  support  is  available? 

What  media  does  the  tool  support? 

With  what  standards  and  specifications  (if  any) 
does  the  tool  comply? 

What  instructional  strategies  does  the  tool  support 
and  how  well? 

What  support  for  analysis  does  the  tool  provide? 

What  student  progress  recording  does  the  tool 
support? 

For  what  development  environment(s)  is  the  tool 
intended? 

What  developer  aids  does  the  tool  provide? 

What  are  the  systems  requirements  for  the  tool? 

How  easy  (or  difficult)  is  the  tool  to  set  up  and  use? 

What  curriculum  structuring  does  the  tool  provide? 

How  extensible  is  the  tool? 

What  interactive  formats  does  (e.g.,  questions  and 
answers)  the  tool  support? 

What  navigation  controls  does  the  tool  provide? 

What  text  options  does  the  tool  support? 

What  graphics  options  does  the  tool  support? 

What  sound  options  does  the  tool  support? 

What  learner  control  options  does  the  tool  support? 

What  natural  language(s)  does  the  tool  support? 

How  are  this  tool  and  its  products  maintained? 
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7.1  SELECTION  ISSUE  #1:  WHAT  DOES  THE  TOOL  DO? 

Where  in  the  Analysis,  Design,  Development,  Implementation,  and  Evaluation  (ADDIE)  cycle  does  the  tool 
fit?  As  may  be  apparent  from  this  report,  most  authoring  capabilities  concentrate  on  development  -  getting  the 
material  out  of  designers’  heads  and  into  the  computer.  Evaluations  in  the  form  of  student  assessment, 
test  construction,  and  student  reporting  receive  less,  but  appreciable,  attention  from  authoring  tools.  Rarely, 
however,  do  these  evaluations  treat  the  design  of  instructional  materials  and  help  identify  instructional 
approaches  that  reliably  lead  to  instructional  outcomes  as  Woolf  and  Regian  [52]  discuss  under  the  heading  of 
instructional  engineering.  Attention  to  implementation  is  lacking,  and  this  inattention  is  unfortunate  because 
of  the  extent  to  which  blended  approaches  of  direct  technology-based  instruction  and  classroom  and  human 
instruction  are  being  used  across  NATO  /  Partner  countries  (e.g.,  [53]).  Also,  finding  authoring  capabilities 
that  deal  with  the  system  issues  of  instructional  analysis  to  any  degree  is  difficult. 

Most  authoring  capabilities  are  developed  on  the  assumption  that  analysis  and  design  have  been  completed 
before  they  are  called  into  use,  and  this  assumption  may  be  sufficient  for  many  users.  However,  the  area  of 
development  has  many  varieties  of  tool  functionalities.  Some  tools  are  designed  to  provide  a  full  suite  of 
development  services,  including  course  unit  mapping  (which  units  fit  where);  a  full  array  of  interaction 
possibilities,  including  a  wide  spectrum  of  multimedia  possibilities;  many  forms  of  answer  processing, 
individualization,  and  branching  in  response  to  student  input;  and  so  forth.  Examples  of  these  types  of  authoring 
tools  are  found  in  such  “full-service”  capabilities  as  CourseBuilder,  Desire2Leam,  and  Authorware.  Other  tools 
focus  on  importing  PowerPoint  slides  or  Word  documents  into  an  instructional  sequence  (Rapid,  Course  Genie), 
some  are  only  concerned  with  learner  assessment  and  the  construction  of  tests  (the  Assessment  Design  and 
Delivery  System  (ADDS)  developed  by  the  Center  for  Research  on  Evaluation,  Standards,  and  Student  Testing 
at  the  University  of  California  at  Los  Angeles  (UCLA)),  and  still  others  focus  on  the  development  of  simulations 
(RealWorld,  DIAG,  ReAct). 

Some  authoring  tools  are  designed  to  operate  with  other  software,  which  may  be  other  authoring  tools 
intended  for  the  development  of  instructional  materials  (e.g.,  DIAG  and  REDEEM)  or  more  general-purpose 
software  (e.g.,  Flash).  In  either  case,  the  tools  are  likely  to  assume  coordinated  application  of  other  software 
packages  with  which  users  are  expected  to  have  some  familiarity  and  facility. 

Finally,  many  authoring  capabilities  do  many  things.  However,  some  things  are  more  easily  done  than  others. 
The  predominate  instructional  approach  planned  for  implementation  should  be  identified  and  then  used  to  test 
candidate  capabilities  to  ensure  that  it  is  not  inordinately  difficult  to  implement  using  one  or  another  of  them. 


7.2  SELECTION  ISSUE  #2:  WHO  WILL  BE  DEVELOPING  THE  INSTRUCTION? 

A  perennial  observation  about  any  development  leads  with  the  following  question:  Do  you  want  the 
development  to  be  done  inexpensively,  quickly,  or  well?  Pick  two. 

Of  course,  we  would  like  to  choose  all  three  options,  but  choosing  just  two  turns  out  to  be  far  more  realistic. 
The  observation  can  be  applied  to  the  first  issue  in  choosing  an  authoring  tool:  Who  will  develop  material 
using  it?  Table  7-2  illustrates,  in  the  context  of  the  question  concerning  cost,  time,  and  quality  (expanded  to 
consider  content,  instructional  design,  and  technological  production),  the  notional  consequences  of  various 
choices  prior  to  selecting  an  authoring  tool. 


7-2 


RTO-TR-HFM-129 


GUIDELINES  FOR  SELECTING  AUTHORING  CAPABILITIES 


Table  7-2:  Likely  Consequences  Arising  from  Choice  of  Developer(s). 


Quality 

Developers 

Inexpensive 

Quick 

Accurate 

Content 

Good 

Instructional 

Design 

Good 

Technological 

Production 

Training 

Technologist 

N 

N 

? 

Y 

Y 

Information 

Technologist 

N 

? 

N 

N 

Y 

Domain 

Specialist 

? 

N 

Y 

N 

N 

Local 

Commands 

Y 

? 

Y 

N 

N 

Authoring 

Teams 

N 

N 

Y 

Y 

Y 

Note  for  Table  7-2:  N  =  Probably  No;  Y  =  Probably  Yes;  ?  =  Unknown. 


Table  7-3  suggests  what  we  can  realistically  hope  would  result  from  the  appropriate  choice  of  authoring 
tool(s).  Some  outcomes  still  remain  uncertain,  but  the  number  of  “probably  no”  judgments  is  down  to  one. 


Table  7-3:  Likely  Consequences  with  Appropriate  Choice  of  Authoring  Tool(s). 


Quality 

Developers 

Inexpensive 

Quick 

Accurate 

Content 

Good 

Instructional 

Design 

Good 

Technological 

Production 

Training 

Technologist 

? 

Y 

? 

Y 

Y 

Information 

Technologist 

N 

Y 

? 

? 

Y 

Domain 

Specialist 

? 

Y 

Y 

? 

Y 

Local 

Commands 

Y 

Y 

Y 

? 

Y 

Authoring 

Teams 

? 

Y 

Y 

Y 

Y 

Note  for  Table  7-3:  N  =  Probably  No;  Y  =  Probably  Yes;  ?  =  Unknown. 
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However  accurate  these  judgments  may  or  may  not  be,  Tables  7-2  and  7-3  emphasize  that  the  choice  - 
or  simply  the  availability  -  of  people  to  do  the  authoring  is  important  and  should  be  considered  up  front  in 
deciding  what  authoring  tools  to  select  for  a  development. 

Questions  along  this  line  might  include  the  following:  What  assumptions  did  the  tool’s  developers  make  about 
its  users?  Is  it  primarily  intended  for  use  by  instructional  technologists,  information  technologists,  SMEs, 
computer  programmers,  or  classroom  instructors?  What  support  does  it  provide  to  these  different  classes  of 
users? 

An  important  issue  is  whether  the  tool  will  be  used  by  individuals  or  teams.  Some  systems  provide  explicit 
aids  for  team  development.  Boot’s  [39]  research  addresses  the  issue  of  using  authoring  tools  to  facilitate 
communication  between  instructional  designers  and  SMEs  who  are  designing  the  materials  and  the  software 
engineers  who  are  implementing  the  designs  in  computer  code.  Boot’s  research  suggests  the  need  to  consider 
the  depth  of  technical  quality  among  the  people  involved  in  design  and  development.  He  found  that  authoring 
tools  could  compensate  somewhat  for  lack  of  expertise  -  but  only  to  a  limited  extent. 


7.3  SELECTION  ISSUE  #3:  WHAT  IS  THE  QUALITY  OF  THE  TOOL? 

Authoring  tools  cannot  be  held  solely  responsible  for  low-quality  instructional  materials  any  more  than  they 
can  be  given  full  credit  for  high-quality  materials.  They  are,  after  all,  tools.  The  choice  of  authoring  tools 
should  include  a  review  of  products  for  which  they  have  been  used  in  the  past.  Assessments  of  these  products 
for  technical  and  instructional  quality  and  for  ease  of  implementation  are  not  always  available  from  empirical 
tests,  but  they  are  highly  desirable  and  should  be  sought. 


7.4  SELECTION  ISSUE  #4:  WHAT  DOES  THE  TOOL  COST? 

Some  questions  about  cost  issues  are  as  follows:  What  licensing  agreements  are  available?  What  enterprise¬ 
wide  licenses,  if  any,  does  the  tool  offer?  What  update  and  maintenance  fees  do  the  vendor  charge  and  on 
what  schedule?  Are  single-user  licenses  available? 


7.5  SELECTION  ISSUE  #5:  HOW  RELIABLE  IS  THE  SUPPLIER  PROVIDING 
THE  TOOL? 

Some  questions  about  the  supplier  are  as  follows:  What  is  the  size  of  the  company?  How  long  has  it  been  in 
operation?  How  many  authors  are  using  the  tool?  What  is  the  experience  level  of  the  company  staff?  What  types 
of  organizations  are  among  its  clients?  What  is  the  company’s  reputation  for  quality,  service,  and  price? 
How  current  is  this  information? 


7.6  SELECTION  ISSUE  #6:  HOW  WIDELY  USED  IS  THE  TOOL? 

The  more  widely  used  the  system,  the  more  likely  that  assistance  will  be  available  and  the  more  likely  that  it 
will  be  supported  in  the  future.  This  issue  suggests  a  bootstrapping  problem  for  tool  developers.  They  may 
have  few  users  because  they  do  not  know  how  to  provide  good  service,  and  their  inadequate  service  prevents 
them  from  having  enough  users  to  show  them  how  to  improve  it. 
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7.7  SELECTION  ISSUE  #7:  WHAT  TECHNICAL  SUPPORT  IS  AVAILABLE? 

A  sufficient  population  of  users  will  support  vigorous  and  knowledgeable  user  groups  that  are  capable  of 
answering  questions  themselves  and  also  can  point  out  improvements  needed  in  the  authoring  tool  and  indicate 
what  priority  these  improvements  deserve.  In  addition  to  active  user  groups,  telephone  assistance  should  also  be 
available  at  all  times,  and  the  assistance  should  be  able  to  reach  back  as  far  as  necessary  to  find  technicians  who 
have  the  knowledge  needed  to  solve  whatever  problems  may  arise.  In  addition,  even  though  user  groups  can 
provide  considerable  training  and  other  resources  needed  by  those  using  the  author  tool,  the  supplier  should 
provide  relevant  and  accessible  training  for  all  levels  of  users.  Finally,  the  technical  and  functional 
documentation  should  be  up-to-date  and  of  high  quality. 


7.8  SELECTION  ISSUE  #8:  WHAT  MEDIA  DOES  THE  TOOL  SUPPORT? 

A  substantial  body  of  research  has  demonstrated  that  visual  (parallel)  and  verbal  (serial)  processes  operate  in  a 
cognitively  additive  and  complementary  fashion  [54].  At  the  most  concrete  level,  pictures  and  speech/text 
operate  together  to  increase  learning  and  transfer  more  than  either  can  in  isolation.  In  brief,  multimedia 
capabilities  enhance  learning,  and  the  authoring  capabilities  that  support  them  will  encourage  better  instructional 
practice. 

Determining  what  media  an  authoring  tool  can  display  to  learners  and  problem  solvers  is  worthwhile. 
Selection  of  a  tool  should  be  keyed  to  the  media  that  designers  plan  to  use.  Text,  graphics,  animation,  video, 
audio,  voice  recognition,  and  external  instrumentation  (e.g.,  light  pens,  track  balls,  and  mouse  clicks),  along 
with  sensors  of  all  sorts,  are  of  interest  in  this  decision. 

Some  questions  might  include  the  following:  What  graphic  and  animation  files  does  the  tool  support 
(e.g.,  Joint  Photographic  Experts  Group  (JPEG),  Graphics  Interchange  Format  (GIF),  Portable  Network 
Graphics  (PNG),  and  Small  Web  Format  (SWF))?  What  and  how  does  it  support  streaming  video  and 
streaming  audio?  What  editing  capabilities  does  it  provide  for  graphics,  automation,  video,  and  audio?  Does  it 
support  simple  interfaces  with  external  hardware,  real-time  control  of  external  hardware,  and/or  overlay  and 
synchronization  of  external  video?  As  with  all  features,  one  should  assess  not  only  if  these  capabilities  are 
available,  but  also  how  easy  they  are  to  use. 


7.9  SELECTION  ISSUE  #9:  WITH  WHAT  STANDARDS  AND  SPECIFICATIONS 
(IF  ANY)  DOES  THE  TOOL  COMPLY? 

Industry-wide  standards  and  specifications  are  critical  in  today’s  authoring  environments  (e.g.,  [55]). 
They  include  the  Sharable  Content  Object  Reference  Model  (SCORM),  Content  Object  Repository  Discovery 
and  Registration/Resolution  Architecture  (CORDRA),  Learning  Object  Metadata  (LOM)  specifications, 
the  Aviation  Industry  Computer-Based  Training  Committee  (AICC)  HTIP-based  AICC/CMI  Protocol  (HACP), 
IMS  Content  Packaging,  IMS  Question  and  Testing  Interoperability,  and  others.  Many  of  the  authoring 
capabilities  reviewed  by  the  Task  Group  are  SCORM  compliant.  The  exceptions  tended  to  be  capabilities 
developed  for  research  purposes. 

Additional  standards  and  specifications  can  be  expected  to  emerge  as  efforts  continue  to  increase  the 
standardization  of  practices  that  enlarge  the  market  for  technology-based  instruction.  It  is  important  to  ask 
with  which,  if  any,  industry  standards  or  specifications  does  the  tool  conform.  It  is  also  important  to  determine 
which  of  these  industry  standards  or  specifications  are  certified  by  the  standard/specification  organization  to 
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be  genuinely  compliant.  The  marketing  enthusiasm  of  vendors  may  lead  them  to  claim  compliance  with  a 
popular  standard  or  specification.  Their  certification  of  compliance  should  be  verified. 


7.10  SELECTION  ISSUE  #10:  WHAT  INSTRUCTIONAL  STRATEGIES  DOES 
THE  TOOL  SUPPORT  AND  HOW  WELL? 

Most  technology-based  instructional  strategies  can  be  broadly  classified  as  drill  practice,  tutorials  (as  in  the 
Keller  /  Crowder  PSI  /  intrinsic  programming  example  discussed  earlier),  simulations,  games,  and  generative 
or  “intelligent”  techniques  (as  in  the  guided  conversations  that  remain  the  aim  of  ITS  development).  Even  the 
most  straightforward  of  these  strategies  -  drill  and  practice  -  may  require  randomization  of  presentations, 
dynamic  pooling  of  items,  mathematical  modeling  of  student  progress,  dynamic  parameter  estimation, 
and  other  ways  of  modeling  the  learner  as,  for  instance,  described  by  Foster  and  Fletcher  [56]. 

Within  these  strategies,  what  branching  capabilities  does  the  tool  support?  Does  it  support  an  unlimited 
number  of  menu  branches  -  branching  based  on  learner  responses,  branching  based  on  elapsed  time,  and/or 
branching  based  on  number  of  tries?  Again,  most  instructional  capabilities  can  be  supported  by  most  general- 
purpose  authoring  tools,  but  some  are  more  easily  supported  than  others.  Selection  of  an  authoring  tool  should 
include  a  consideration  of  the  most  effective  and  efficient  strategy  to  achieve  the  targeted  instructional 
objectives  and  the  ease  or  difficulty  with  which  it  supports  that  strategy. 


7.11  SELECTION  ISSUE  #11:  WHAT  SUPPORT  FOR  ANALYSIS  DOES  THE 
TOOL  PROVIDE? 

Does  the  tool  provide  automated  data  capture  and  storage?  What  statistical  analyses  techniques  does  it 
support?  To  what  and  in  what  detail  does  it  capture  and  record  student  responses?  Does  it  include  both 
keyboard  and  click  stream  data  input  and  latencies?  Does  it  support  item  analyses,  item  characteristic  curves, 
and  analyses  of  simulations  and  simulation  scenarios?  If  the  tool  does  not  support  the  analyses  needed  by  a 
program  of  technology-based  instruction,  separate  computer  programs  to  provide  these  analyses  will  have  to 
be  purchased  or  developed. 


7.12  SELECTION  ISSUE  #12:  WHAT  STUDENT  PROGRESS  RECORDING  DOES 
THE  TOOL  SUPPORT? 

Can  the  tool  easily  generate  reports  on  individual  students  or  students  organized  into  groups  specified  by 
instructors  and  administrators  according  to  project  groups,  classrooms,  academic  discipline,  demographic 
data,  and  so  forth?  If  it  supports  these  groupings,  what  specific  data  can  it  report?  Can  these  data  be  linked  to 
instructional  objectives  in  ways  that  are  useful  to  educational  analysts,  administrators,  and  classroom 
teachers?  What  capacities  for  collecting,  storing,  and,  especially,  protecting  individual  learner  data  does  the 
tool  support? 


7.13  SELECTION  ISSUE  #13:  FOR  WHAT  DEVELOPMENT  ENVIRONMENT(S) 
IS  THE  TOOL  INTENDED? 

Is  the  tool  intended  for  use  only  in  a  stand-alone  environment  or  does  it  include  provisions  for  networked, 
collaborative  development? 
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7.14  SELECTION  ISSUE  #14:  WHAT  DEVELOPER  AIDS  DOES  THE  TOOL 
PROVIDE? 

For  instance,  does  the  tool  provide  spell  checking,  thesaurus  access,  a  “What  You  See  Is  What  You  Get” 
(WYSIWYG)  interface,  drop  and  drag  objects,  and  instructional  content  and  sequence  tracing?  Does  it  provide 
quick-preview  functions?  Can  it  print  content  for  learners  and/or  authors? 


7.15  SELECTION  ISSUE  #15:  WHAT  ARE  THE  SYSTEM  REQUIREMENTS  FOR 
THE  TOOL? 

This  issue  is  one  of  the  most  important  in  selecting  authoring  capabilities.  What  hardware  and  system  software 
are  required  for  authors  to  use  the  tool?  What  hardware  and  system  software  are  required  for  learners  and 
users  to  apply  the  tool’s  products?  What  are  the  minimum  capabilities  required?  Beyond  these  minimums, 
what  capabilities  and  capacities  are  recommended  by  the  tool  providers? 


7.16  SELECTION  ISSUE  #16:  HOW  EASY  (OR  DIFFICULT)  IS  THE  TOOL  TO 
SET  UP  AND  USE? 

For  what  operating  environment  is  the  tool  intended?  Does  it  require  learning  management  systems,  learning 
content  management  systems,  or  specific  instances  of  these  to  function?  Does  it  require  plug-ins  and/or 
software  capabilities  that  must  be  obtained  separately? 


7.17  SELECTION  ISSUE  #17:  WHAT  CURRICULUM  STRUCTURING 
CAPABILITIES  DOES  THE  TOOL  PROVIDE? 

Does  it  provide  overall  course,  module,  and/topic  structuring  capabilities  such  as  tree  structures,  storyboards, 
or  concept  maps?  Does  it  integrate  the  structure  with  ongoing  activities  by  designers  and  developers  to 
provide  advice,  reminders,  and/or  review  of  development  progress? 


7.18  SELECTION  ISSUE  #18:  HOW  EXTENSIBLE  IS  THE  TOOL? 

What,  if  any,  computer  or  scripting  language(s)  does  the  tool  support  or  require?  How  and  how  easily  can 
authors  develop  their  own  templates? 


7.19  SELECTION  ISSUE  #19:  WHAT  INTERACTIVE  FORMATS  (E.G., 
QUESTIONS  AND  ANSWERS)  DOES  THE  TOOL  SUPPORT? 

Does  the  tool  support  (perhaps  with  templates  and/or  content  wizards)  formats  such  as  true/false,  multiple 
choice,  short  answer,  fill-in-the-blank,  essays  (graded  by  an  instructor  or  by  computer),  matching,  sequence, 
hot-spot,  drag  and  drop?  Does  it  provide  a  means  to  align  displayed  objects,  place  objects  precisely, 
synchronize  different  display  modalities  (e.g.,  appearance  of  visual  objects  with  accompanying  audio), 
and  hyperlink  objects  to  other  resources?  Does  the  tool  support  presenting  questions  in  random  order, 
presenting  multiple  choice  distracters  in  random  order,  presenting  questions  drawn  from  a  pool,  automatically 
stored  student  responses,  turning  feedback  on  and  off,  randomized  feedback,  timed  questions,  timed  tests, 
tests  that  automatically  select  content  based  on  learner  performance,  and  branching  based  on  responses? 
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7.20  SELECTION  ISSUE  #20:  WHAT  NAVIGATION  CONTROLS  DOES  THE 
TOOL  PROVIDE? 

Does  its  standard  interface  include  controls  such  as  “next,”  “back,”  “exit,”  “view  menu,”  “glossary,”  “help,” 
and  “bookmark”?  Does  the  tool  support  radio  buttons,  pull-down  menus,  list  boxes,  scroll  bars,  and  dialog  box 
frames? 


7.21  SELECTION  ISSUE  #21:  WHAT  TEXT  OPTIONS  DOES  THE  TOOL 
SUPPORT? 

Does  it  support  variable  fonts,  sizes,  and  colors,  variable  styles  (bold,  underline,  italic,  and  so  forth),  variable 
formats  (centering),  importing  of  text,  and  search  and  replace? 


7.22  SELECTION  ISSUE  #22:  WHAT  GRAPHICS  OPTIONS  DOES  THE  TOOL 
SUPPORT? 

Does  it  support  a  variety  of  objects  (line,  box,  shapes,  and  so  forth),  high-resolution  color,  animation  options, 
graphic  scaling  and  rotating,  bitmapped  graphics,  object-oriented  graphics,  graphics  and/or  text  overlay, 
and  screen  capture? 


7.23  SELECTION  ISSUE  #23:  WHAT  SOUND  OPTIONS  DOES  THE  TOOL 
SUPPORT? 

Does  it  support  computer-generated  sounds,  synthesized  speech,  3D  audio,  and  digitized  audio  editing  and  play? 

7.24  SELECTION  ISSUE  #24:  WHAT  LEARNER  CONTROL  OPTIONS  DOES 
THE  TOOL  SUPPORT? 

Does  it  support  learner  control  of  sequence,  programmable  learner  control  keys,  learner  restart  options,  learner 
bookmarking,  learner  access  to  dictionary  or  glossaries,  learner  access  to  note-taking  facilities,  and  learner 
access  to  printing  of  selected  information? 


7.25  SELECTION  ISSUE  #25:  WHAT  NATURAL  LANGUAGE(S)  DOES  THE 
TOOL  SUPPORT? 

What  natural  (i.e.,  human)  languages  and  character  sets  does  it  support?  This  question  is  particularly 
important  for  NATO  and  Partner  developers.  It  applies  to  reference  and  training  manuals  (on-line  and/or 
off-line),  to  controls  and  statements  used  by  the  tool,  and  to  features  such  as  spelling  and  grammar  checkers. 
Even  though  the  learning  materials  will  be  presented  in  one  language  (e.g.,  English),  production-support 
materials  should  also  be  provided  in  languages  that  are  more  familiar  to  the  designers  and  developers  of  the 
materials. 
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7.26  SELECTION  ISSUE  #26:  HOW  ARE  THIS  TOOL  AND  ITS  PRODUCTS 
MAINTAINED? 

A  not-uncommon  problem  is  that  instructional  materials  cannot  not  be  maintained  or  updated  after  a  period  of 
time  because  the  tools  or,  more  frequently,  the  versions  of  the  software  used  to  develop  these  tools  are  no  longer 
available.  What  solutions  does  the  tool  provider(s)  offer  for  this  problem?  What  version  and  configuration 
management  do  the  tool  provider(s)  offer? 
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Authoring  capabilities  have  progressed  substantially  since  the  days  of  the  SAGE  System  Training  Program, 
COURSEWRITER,  and  TUTOR.  These  new  capabilities  have  become  increasingly  agile,  easy  to  use, 
wide  ranging  in  their  coverage  of  instructional  approaches,  and  automated.  They  can  now  be  used  by  many 
people  -  especially  those  who  do  not  have  substantial  experience  in  computer  software  technology  - 
to  produce  high-quality  technology-based  instructional  materials  rapidly  and  inexpensively.  Thanks  to  several 
standardization  efforts,  these  capabilities  can  generate  material  that  is  portable  across  many  platforms,  durable 
despite  frequently  introduced  modifications  of  underlying  system  software,  and  reusable. 

A  significant  portion  of  authoring  capabilities  will  probably  evolve  in  the  direction  of  peer  production,  thanks 
to  the  rapidly  growing  marketplace  for  Web  2.0  materials.  The  goal  of  fully  automated  (generative  and 
on-demand  production)  systems  remains  elusive,  but  progress  has  been  made  in  using  commercial  databases, 
such  those  from  CAD  for  equipment  operation  and  maintenance  and  those  from  professional  simulation 
vendors  for  in-theater  training  and  mission  rehearsal,  to  generate  realistic  learning  environments.  Finally, 
the  body  of  empirical  data  to  demonstrate  the  ability  of  authoring  capabilities  to  reduce  costs,  time,  and  other 
resources  in  the  production  of  education,  training,  and  performance  aiding  environments  is  growing  in  quantity 
and  quality.  Evidence  of  cost  effectiveness  to  inform  trade-off  decisions  is  still  needed,  but  it,  too,  is  being 
produced,  albeit  erratically  and  slowly.  Nonetheless,  the  value  of  authoring  capabilities  seems  sufficiently 
demonstrated  by  current  data  and  user  acceptance  to  ensure  its  continued  development  and  evolution. 
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I.  BACKGROUND  AND  JUSTIFICATION  (RELEVANCE  TO  NATO) 

From  1985  through  2003,  NATO  has  held  workshops  on  technologies  associated  with  military  training. 
These  workshops  covered  Computer-Based  Instruction,  Training  System  Design,  Distributed  Training, 
and  Virtual  Reality.  While  technologies  have  dramatically  changed,  many  of  the  problems  and  issues  related  to 
training  technologies  -  its  application,  acceptance,  and  effectiveness  -  are  still  relevant  today.  The  restructuring 
of  the  military  forces  in  size  and  composition,  along  with  decreased  funds  for  real  exercises,  has  continued  to 
focus  attention  on  how  to  maintain  a  well-trained  and  flexible  force.  Recent  changes  in  NATO  and  the  European 
infrastructure  have  stimulated  the  Research  Technology  Organization  (RTO)  to  sponsor  an  October  2003 
symposium  that  focused  on  technologies  applicable  to  improving  military  training. 

Military  training  must  prepare  individuals  to  enter  into  harm’s  way  and  perform  physically  and  mentally 
demanding  tasks  at  the  highest  possible  levels  of  proficiency  in  a  time-critical  manner.  This  requirement  may 
be  the  defining  characteristic  of  military  training.  With  advanced  technology  available  to  any  buyer,  training 
can  mean  the  difference  between  success  and  failure,  life  and  death.  Such  training  must  be  dynamic,  allowing 
continual,  just-in-time  modification  and  updating  -  especially  when  it  is  to  be  distributed  anytime  and 
anywhere  it  is  needed.  Meeting  this  requirement  requires  authoring  capabilities  for  the  rapid  development  of 
training  materials  that  are  maximally  responsive  to  changing  operational  requirements,  adaptive  to  cultural 
differences,  and  readily  used  by  military  personnel.  This  technical  tool  will  be  especially  important  for 
preparing  NATO  Response  Force  (NRF)  command  personnel  by  computer-mediated,  just-in-time  training 
systems  that  will  be  the  standard  training  and  mission  rehearsal  tools  of  the  future. 


II.  OBJECTIVE(S) 

This  Exploratory  Team  (ET)  will  focus  on  recommendations  made  at  the  conclusion  of  the  October  2003 
symposium  in  Genoa,  Italy.  These  recommendations  concern  the  use  of  technologies  to  develop  computer- 
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mediated  training  environments  quickly  and  effectively.  This  focus  suggests  that  the  ET  analyze  the  value  and 
opportunities  provided  by  advanced  and  emerging  technology-based  authoring  systems.  The  ET  will  use  the 
analysis  to  formulate  a  TAP  and  Terms  of  Reference  (TOR)  for  a  Human  Factors  and  Medicine  (HFM)  Task 
Group.  One  element  of  the  TOR  for  the  Task  Group  will  be  the  scheduling  of  a  “Cooperative  Demonstration  of 
Technology”  (CDT)  and/or  a  Specialists’  Workshop  as  the  capstone  events  for  the  work  of  the  Task  Group. 


III.  TOPICS  TO  BE  COVERED 

•  State-of-the-art  of  authoring  systems  and  its  uses  in  NATO  (e.g.,  for  selection  of  curriculum  content, 
selection  of  instructional  approaches,  design  of  instruction,  efficient  development  of  materials,  efficient 
achievement  of  targeted  instructional  outcomes,  contributions  to  operational  effectiveness); 

•  Measuring  the  costs  and  effectiveness  of  authoring  systems  as  these  systems  contribute  to  improved 
human  performance; 

•  Guidelines  (including  what  works)  for  operational  application  of  current  and  advanced  authoring  systems; 

•  Else  of  simulation  (including  game  technologies)  in  the  authoring  process;  identifying  where  and  what  type 
of  content  it  can  be  most  effective/efficient  for  authoring; 

•  Automated  and/or  ‘intelligent’  capabilities  for  authoring; 

•  Sharable,  reusable  instructional  objects  used  in  authoring  and  examples  of  the  cost/value  of  interoperability; 

•  Authoring  of  “stories”  for  adaptable  case-based  learning  environments;  and 

•  Authoring  of  performance  aiding  (as  well  as  instructional)  capabilities. 


IV.  DELIVERABLE  AND/OR  END  PRODUCT 

Report  to  the  HFM  Panel  by  the  Spring  2005  meeting  on  the  feasibility  of  establishing  a  Task  Group. 


V.  TECHNICAL  TEAM  LEADER  AND  LEAD  NATION 

Dr.  Dexter  Fletcher,  Institute  for  Defense  Analyses  (IDA)  (01)  703.578.2837,  fletcher@ida.org,  EISA. 


VI.  NATIONS  WILLING  TO  PARTICIPATE 

Initial  membership  will  be  invited  from  the  program  committee  of  the  recent  training  symposium  and  from  the 
recently  completed  Advanced  Distributed  beaming  (ADF)  Task  Group.  These  nations  include  Belgium,  Canada, 
the  Czech  Republic,  France,  Germany,  Italy,  Netherlands,  the  United  Kingdom,  and  the  United  States. 


VII.  NATIONAL  AND/OR  NATO  RESOURCES  NEEDED  (PHYSICAL  AND  NON¬ 
PHYSICAL  ASSETS) 

Access  to  representative  software  currently  being  used  for  authoring;  access  to  knowledgeable,  representative 
users  of  authoring  systems;  ability  to  participate  in  teleconferences  and  email  to  reduce  the  need  for  face-to-face 
meetings;  and  willingness  to  experiment  with  advanced  authoring  techniques  to  assess  technical  opportunities, 
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international  and  cultural  issues,  Research  and  Development  (R&D)  priorities  and  requirements,  and  NATO 
operational  priorities  and  requirements. 


VIII.  RESEARCH  AND  TECHNOLOGY  AGENCY  (RTA)  RESOURCES  NEEDED 
(E.G.,  CONSULTANT  FUNDING) 

Three  consultant  trips  tied  to  meetings  at  locations  to  be  identified. 
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I.  ORIGIN 

A.  Background 

This  Task  Group  (TG)  responds  to  recommendations  made  at  the  conclusion  of  the  Human  Factors  and 
Medical  Panel  Symposium  on  Advanced  Technologies  for  Military  Training  held  in  Genoa,  Italy,  in  October 
2003.  These  recommendations  concern  the  use  of  technologies  to  develop  computer- mediated  training 
environments  quickly  and  effectively.  This  Exploratory  Team  (ET)  is  to  review  the  value  and  opportunities 
provided  by  advanced  and  still-emerging  technologies  for  creating  such  environments  and  suggest  their 
further  development  focused  on  NATO  requirements  for  future  training  capabilities. 

As  reviewed  in  many  NATO  studies,  the  case  (based  on  empirical  data)  for  computer-mediated  learning 
environments  can  be  summarized  from  the  literature  as  follows: 

•  Tailoring  instruction  (education  and  training)  to  the  needs  of  individual  students  has  been  found  to 
increase  learning  outcomes  by  as  much  as  two  standard  deviations,  which  is  roughly  equivalent  to 
raising  the  achievement  of  50th  percentile  students  to  that  of  98th  percentile  students.  However, 
requiring  one  (human)  instructor  for  every  student  is  an  economic  impossibility. 

•  Computer  technology  can,  in  many  cases,  make  this  proven  instructional  capability  affordable.  Under 
any  appreciable  student  load,  it  is  much  more  cost  effective  to  deliver  instruction  using  technology 
than  to  provide  a  (human)  tutor  for  each  student.  Such  technology  supplies  much  of  the  tailoring 
(or  individualization)  of  instruction  that  accounts  for  the  success  of  individual  tutorial  instruction. 

•  On  the  basis  of  empirical  evidence,  technology-based  instruction  has  been  found  to  be  more  effective 
than  current  classroom  instructional  approaches  in  many  settings  across  many  subject  matters. 

•  Technology-based  instruction  is  generally  less  costly  than  current  instructional  approaches,  especially 
when  many  students  are  to  be  trained  or  when  instructional  objectives  require  the  operation  or 
maintenance  of  costly  equipment. 

•  Technology-based  instruction  has  been  found  over  many  evaluation  studies  to  decrease  the  time 
needed  to  reach  targeted  instructional  objectives  by  about  30%,  thereby  preparing  personnel  more 
quickly  for  operational  duty  assignments  and  reducing  infrastructure  costs. 

•  Remote  and  massive  digital  storage  technology,  combined  with  global  communication  capabilities, 
makes  possible  the  delivery  of  technology-based  education,  training,  and  performance  aiding  anytime, 
anywhere. 

In  brief,  a  “Rule  of  Thirds”  emerges  from  assessments  of  computer-mediated  instruction.  That  is,  empirical 
findings  suggest  that  the  use  of  computer-mediated  instruction  reduces  the  costs  of  instruction  by  about  one- 
third  and  either  reduces  time  of  instruction  by  about  one-third  or  increases  the  amount  of  skills  and  knowledge 
acquired  by  about  one-third. 

Given  this  potential,  it  is  not  surprising  to  find  strong  international  interest  in  the  production  and  use  of 
computer-mediated  learning  environments  to  enhance  NATO  operational  capabilities. 
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B.  Justification 

NATO  interest  in  technology-based  instruction  does  not  arise  solely  from  cost  efficiencies  and  technological 
opportunities.  Restructuring  of  military  forces  in  size  and  composition,  along  with  decreased  funding  and 
environmental  tolerance  for  field  exercises  and  practice,  has  substantially  increased  requirements  for  training 
capabilities  such  as  those  offered  by  computer-mediated  environments.  These  environments  can  help  establish 
and  maintain  well-trained,  agile  forces  that  ensure  commanders  ready  access  to  the  human  performance  and 
competence  that  are  vital  for  operational  success.  In  addition,  they  allow  forms  of  training  that  for  reasons  of 
cost,  safety,  repeatability,  and  accessibility  cannot  be  provided  by  any  other  means. 

Military  education,  training,  and  performance  aiding  must  prepare  individuals  to  enter  into  harm’s  way  and 
perform  physically  and  mentally  demanding  tasks  at  the  highest  possible  levels  of  proficiency  in  a  time-critical 
manner.  This  requirement  may  be  the  defining  characteristic  of  military  pedagogy.  With  advanced  technology 
available  to  any  buyer,  pedagogy  and  preparation  can  mean  the  difference  between  success  and  failure,  life  and 
death.  The  necessary  pedagogy  must  be  dynamic,  allowing  continual,  just-in-time  modification  and  updating  of 
training  environments  and  materials  -  especially  when  these  materials  are  to  be  distributed  anytime  and 
anywhere  they  are  needed.  Meeting  this  requirement  requires  capabilities  for  the  rapid  development  of  training 
materials  and  systems  that  are  maximally  responsive  to  changing  operational  requirements,  readily  adapted  to 
cultural  differences,  and  easily  used  by  military  personnel. 

Technological  opportunities  that  promise  to  both  lower  the  cost  and  enhance  the  speed  and  agility  of  preparation, 
or  “authoring,”  for  computer-mediated  education,  training,  and  performance  aiding  have  been  developed  and 
continue  to  emerge.  These  opportunities  include  authoring  templates;  authoring  that  uses  pre-stored,  reusable, 
sharable  instructional  objects,  simulation  and  gaming-based  authoring;  and  “intelligent”  automated  authoring 
techniques.  The  ability  of  these  technologies  to  enhance  the  agility  and  responsiveness  of  training  environments 
in  meeting  rapidly  emerging  and  often  unforeseen  exigencies  of  modem  military  operations  will  save  lives  and 
substantially  enhance  the  human  competence  available  to  NATO  commanders. 

II.  OBJECTIVES 
A.  Scope 

The  Task  Group  will  review  and  assess: 

•  The  state-of-the-art  of  authoring  systems  (e.g.,  for  selection  of  curriculum  content,  selection  of 
instructional  approaches,  design  of  instruction,  efficient  development  of  materials,  efficient  achievement 
of  targeted  instructional  outcomes,  contributions  to  operational  effectiveness); 

•  Authoring  as  it  is  currently  done  in  the  NATO  alliance  -  including  the  identification  and  presentation 
of  realistic  examples,  relevant  time  constraints,  and  other  considerations  that  impact  the  ability  to 
develop  learning  environments  and  share  their  contents; 

•  Measures  of  the  costs  and  effectiveness  of  authoring  systems  as  they  contribute  to  improved  operational 
performance; 

•  Guidelines  for  operational  application  of  current  and  advanced  authoring  systems; 

•  Use  of  simulation  and  gaming  technologies  as  tools  in  the  authoring  process; 

•  Automated  and/or  “intelligent”  capabilities  for  authoring; 

•  Sharable,  reusable  instructional  objects  used  in  authoring;  and 

•  Authoring  of  performance  aiding  (as  well  as  instructional)  capabilities. 
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The  goals  of  the  ET  will  be  to  describe  current  capabilities  and  limitations  and  then  impact  on  NATO 
operations.  The  ET  is  intended  to: 

•  Increase  within  the  NATO  community  the  awareness  of  current  and  emerging  authoring  capabilities, 
costs,  and  effectiveness; 

•  Identify  requirements  and  capabilities  of  authoring  systems  and  the  computer-mediated  environments 
they  produce  for  usability  and  interoperability  across  the  NATO  nations; 

•  Identify  and  prioritize  demonstrations,  experimentation,  and  research  needed  to  enhance  current 
authoring  capabilities  in  areas  of  particular  importance  in  meeting  the  needs  of  NATO  operations; 

•  Propose  a  plan  and  continuing  process  for  implementing  current  and  emerging  capabilities  to  ensure 
that  they  provide  maximum  and  timely  benefit  to  the  NATO  community;  and 

•  Review  what  other  Research  Technology  Organisation  (RTO)  Panels  and  Groups  are  doing  and 
opportunities  that  their  activities  might  offer  for  inter-Panel  coordination  and  cooperation. 

C.  Products 

Consistent  with  its  goals,  the  products  of  the  ET  will  include: 

•  Briefings; 

•  Documentation  of  existing  and  evolving  capabilities  of  authoring  systems  and  lessons  learned  in  using 
them; 

•  A  prioritized  research  agenda  for  further  Research  and  Development  (R&D)  of  authoring  systems; 

•  A  plan  for  the  implementation  and  use  of  authoring  systems  within  the  NATO  community;  and 

•  Identification  of  requirements  for  multi-national  interoperability  among  authoring  systems  and  the 
computer-mediated  environments  they  produce. 

D.  Duration 

The  ET  will  operate  for  3  years. 

III.  RESOURCES 

A.  Membership 

Membership  will  include  national  experts  in  authoring  and  instructional  systems  design,  Advanced  Distributed 
Learning  (ADL),  training  simulation  for  individual  and  collective  training,  computer-mediated  instruction, 
NATO  operational  requirements,  training  development  and  implementation,  and  development  and  delivery 
costs  of  technology-based  training  and  performance  aiding. 

The  ET  is  expected  to  include  participation  from  Bulgaria,  Canada,  Denmark,  Germany,  Hungary,  Italy, 
Netherlands,  Norway,  Sweden,  the  United  Kingdom,  and  the  United  States.  Further  participation  from  nations 
and  from  NATO  organisations  is  being  sought.  Task  Group  members  will  include  serving  military  personnel 
and  national  experts  in  simulation  technology,  behavioural  science,  computer  science,  education  and  training, 
and  development  and  use  of  computer-mediated  environments. 
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The  Human  Factors  Medical  Panel  has  selected  the  United  States  as  the  Lead  Nation.  The  United  States  has 
designated  as  Team  Leader  Dr.  Dexter  Fletcher  of  the  Institute  for  Defense  Analyses  (IDA). 

B.  National  and/or  NATO  Resources  Needed 

The  work  of  the  Task  Group  will  build  on  national  programmes.  Additional  resources  required  will  include 
support  for  team  members,  with  funding  for  travel,  for  participation  in  working  meetings,  and  for  such 
preparatory  work  as  is  agreed. 

Access  to,  and  use  of,  education,  training,  and  performance  assets  in  nations  and  between  nations  would  also 
be  required  to  prepare  for  and  then  conduct  the  proposed  multi-national  demonstration  of  the  potential  of 
mission  training  through  distributed  simulation. 

NATO  support  is  requested  for  this  Task  Group.  It  should  include  access  to  NATO  education,  training, 
and  performance  aiding  assets  needed  to  facilitate  cooperative  efforts  and  agreements  between  nations  and  to 
provide  operational  military  staff  as  needed. 

C.  Research  and  Technology  Agency  (RTA)  Resources  Needed 

Funding  will  be  sought  from  the  Cooperative  Planning  Programme. 

Support  to  establish  a  website  will  be  requested. 

Access  to  representative  software  currently  being  used  for  authoring;  access  to  knowledgeable,  representative 
users  of  authoring  systems;  the  ability  to  participate  in  teleconferences  and  email  to  reduce  the  need  for  face- 
to-face  meetings;  and  the  willingness  to  experiment  with  advanced  authoring  techniques  to  assess  technical 
opportunities,  international  and  cultural  issues,  R&D  priorities  and  requirements,  and  NATO  operational 
priorities  and  requirements. 


IV.  SECURITY  CLASSIFICATION  LEVEL 

The  intent  of  the  Task  Group  is  to  operate  at  the  NATO  Unclassified  level. 


V.  PARTICIPATION  BY  PARTNER  NATIONS 

Partner  nations  are  invited  to  participate  in  the  Task  Group. 


VI.  LIAISON 

Contact  and  collaboration  will  be  sought  with  other  NATO  bodies  and  Panels. 
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Boot,  E.  and  Kitchen,  D.  (2005).  “A  Model  for  Selecting  Authoring  Technologies”,  Unpublished  White  Paper 
Prepared  for  NATO  Research  Task  Group  E1FM-129. 

Forrester,  B.  and  Verdon,  J.  (2009).  “Introducing  Peer  Production  into  the  Department  of  Defense”,  Unpublished 
White  Paper  Prepared  for  NATO  Research  Task  Group  HFM-129. 

Kaufman,  D.  and  Funceford,  W.H.  (2009).  “RealWorld:  Simulation  Without  Programmers  (Report  NS  D-3955)”, 
Alexandria,  VA,  USA:  Institute  for  Defense  Analyses. 

NATO  Task  Group  HFM-129  Presentations  (September  2005),  Fos  Angeles,  CA,  USA  (Compact  Disc). 

NATO  Task  Group  HFM-129  Presentations  (May  2006),  Soesterberg,  NFD  (Compact  Disc). 

NATO  Task  Group  HFM-129  Presentations  (September  2006),  Trenchard  Fines,  Upavon,  Wiltshire,  GBR, 
(Compact  Disc). 

NATO  Task  Group  HFM-129  Tutorial  on  Authoring  Technologies,  Presentation,  ITEC  Annual  Meeting,  April 
2007,  Cologne,  Germany. 

NATO  Task  Group  HFM-129  Tutorial  on  Authoring  Technologies,  Presentation,  I/ITSEC  Annual  Convention, 
December  2008,  Orlando,  Florida,  USA 
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The  Task  Group  (TG)  met  four  times: 

1)  20  -  23  September  2005,  University  of  California  at  Los  Angeles  (UCLA),  United  States. 

2)  2-4  May  2006,  Nederlandse  Organisatie  voor  Toegepast  Natuurwetenschappelijk  Onderzoek  (TNO) 
(Netherlands  Organization  for  Applied  Scientific  Research),  Soesterberg,  Netherlands. 

3)  13-16  November  2006,  Trenchard  Lines,  Upavon,  Wiltshire,  United  Kingdom. 

4)  4-6  April  2007,  Cologne,  Germany  in  conjunction  with  ITEC  2007. 

Discussions  and  presentations  of  the  Task  Group  activities  and  findings  have  been  presented  to  the  NATO 
Training  Group  (NTG)  Joint  Services  Sub-Group  (JSSG)  Working  Group  on  Individual  Training  and  Educational 
Developments  at  the  following  meetings: 

•  April  2005,  Copenhagen,  Denmark. 

•  May  2007,  Oberammergau,  Germany. 

•  September  2007,  Bergen,  Norway. 

•  September  2008,  Vienna,  Austria. 

Other  presentations  of  Task  Group  findings: 

•  Authoring  Tools  Workshop,  Fedex  Institute  of  Technology,  University  of  Memphis,  Memphis,  TN, 
USA,  November  2005. 

•  Aviation  Industry  Computer-Based  Training  Committee  Meeting,  San  Diego,  CA,  USA,  February  2006. 

•  ITEC  Annual  Meeting,  April  2007,  Cologne,  Germany,  3 -hour  seminar  -  90  minutes  on  findings  and 
90  minutes  of  hands-on  demonstrations. 

•  Interservice/Industry  Training,  Simulation  and  Education  Conference  (I/ITSEC)  Annual  Convention, 
December  2008,  Orlando,  FL,  USA,  90-minute  tutorial  on  Task  Group  findings  and  recommendations. 
I/ITSEC  presentation  slides  are  attached  to  this  report. 

In  assessing  the  current  capabilities  and  opportunities  offered  by  the  state-of-the-art,  the  Task  Group  reviewed 
presentations  and  demonstrations  of  the  following  authoring  tools: 

•  BBN  Technologies  authoring  for  DARWARS  Ambush!  Provides  rapid  and  flexible  authoring  by 
local  command  staff  for  convoy  simulation  training. 

•  Boxer  Technologies  (now  a  member  of  the  Edvantage  Group)  on  CourseBuilder.  Full  enterprise 
system  -  web-based  and  SCORM  compatible  using  templates  and  drop  and  drag  interfaces. 

•  Behavioral  Technology  Laboratories  (BTL)  /  Center  for  Cognitive  Technology  (CCT)  /  University  of 
Southern  California  (USC)  on  iRides,  ReACT,  Personal  Knowledge  Source  (PKS),  DIAG.  These  are 
all  authoring  tools  providing  ease  of  use  and  rapid  development  of  simulations  and  graphical  content 
in  them. 

•  Carnegie  Mellon  University  (CMU)  on  Cognitive  Tutor  Authoring  Tools  (CTAT).  Developed  by 
CMU  as  a  set  of  authoring  tools  for  Intelligent  Tutoring  Systems  (ITS). 
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•  Desire2Leam.  A  full  enterprise  system  using  open  standards  (Sharable  Content  Object  Reference 
Model  (SCORM)  2004,  IMS,  IEEE  Learning  Object  Metadata  (LOM),  and  so  forth)  and  interoperable 
object  repositories. 

•  Epic  on  Rapid.  Template-driven  system  intended  to  provide  both  simple  and  rapid  authoring 
capabilities,  often  involving  conversion  from  PowerPoint  slides. 

•  Horizon  Wimba  on  Course  Genie.  Rapidly  and  automatically  generates  SCORM-conformant,  online 
courseware  from  Microsoft  Word  and  PowerPoint  sources. 

•  Line  Communications  on  Mobile  Delivery  and  Template  Driven  Authoring. 

•  Micromedia  on  Flash,  Authorware,  Captivate,  and  Breeze.  Full-enterprise  solutions  for  rapid 
development  of  materials  and  maintaining  content  management  through  open  standards. 

•  Reload  Editor.  Provides  a  drop  and  drag  approach  to  packaging  content  so  that  it  conforms  to  SCORM 
2004  specifications. 

•  Royal  Netherlands  Army  on  Flash  Template  Editor.  Intended  to  imbue  media  assets  with  interaction, 
reusability,  and  didactical  features  through  the  use  of  adjustable  templates. 

•  UCLA  /  National  Center  for  Research  on  Evaluation,  Standards,  and  Student  Testing  (CRESST)  on 
Assessment  Design  and  Delivery  System  (ADDS).  Tools  for  authoring  student  assessments. 

•  University  of  Nottingham  on  Reusable  Educational  Design  Environment  and  Engineering  Methodology 
(REDEEM).  Tool  to  minimize  authoring  time  and  effort  by  turning  already  developed  content  into 
pedagogically  useful  learning  environments. 

•  University  of  Twente  on  Learn  Exact.  Provides  templates  for  learning  objects,  metadata,  and  packaging. 

•  USC  /  Information  Sciences  Institute  (ISI)  /  Center  for  Advanced  Research  in  Technology  for  Education 
(CARTE)  on  Tactical  Language  and  Culture  Tutor  (TLCT).  Developed  for  the  Defense  Advanced 
Research  Projects  Agency  (DARPA)  by  USC’ s  CARTE  to  provide  authoring  for  game-  and  simulation- 
based  learning. 

•  VCom3D  on  Vcommunicator,  Gesture  Builder,  Scenario  Authoring  Tool.  Authoring  tools  for 
developing  and  synchronizing  the  actions  (e.g.,  speech,  gestures,  expressions)  of  avatars. 
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